Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
2026-05-04T13:55:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-04T13:55:52Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260504 13:55:53.006594 9147 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
/tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.254:36145
--webserver_interface=127.8.238.254
--webserver_port=0
--builtin_ntp_servers=127.8.238.212:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.238.254:36145,127.8.238.253:36337,127.8.238.252:44781
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260504 13:55:53.302028 9156 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260504 13:55:53.302681 9156 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260504 13:55:53.303125 9156 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260504 13:55:53.335513 9156 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260504 13:55:53.335878 9156 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260504 13:55:53.336155 9156 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260504 13:55:53.336436 9156 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260504 13:55:53.372637 9156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.238.212:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.238.254:36145,127.8.238.253:36337,127.8.238.252:44781
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.254:36145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.238.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 04 May 2026 13:43:23 UTC on e7f111948823
build id 11743
TSAN enabled
I20260504 13:55:53.374065 9156 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 13:55:53.375711 9156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 13:55:53.386473 9162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:55:53.387913 9163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:55:54.579847 9164 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1190 milliseconds
W20260504 13:55:54.580713 9165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:55:54.580706 9156 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260504 13:55:54.584918 9156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260504 13:55:54.588469 9156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260504 13:55:54.590018 9156 hybrid_clock.cc:648] HybridClock initialized: now 1777902954589968 us; error 45 us; skew 500 ppm
I20260504 13:55:54.591257 9156 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 13:55:54.599995 9156 webserver.cc:492] Webserver started at http://127.8.238.254:38403/ using document root <none> and password file <none>
I20260504 13:55:54.601374 9156 fs_manager.cc:362] Metadata directory not provided
I20260504 13:55:54.601691 9156 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 13:55:54.602341 9156 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 13:55:54.609297 9156 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/data/instance:
uuid: "806407748d564439807fe5d73db9abda"
format_stamp: "Formatted at 2026-05-04 13:55:54 on dist-test-slave-f9wt"
I20260504 13:55:54.610819 9156 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/wal/instance:
uuid: "806407748d564439807fe5d73db9abda"
format_stamp: "Formatted at 2026-05-04 13:55:54 on dist-test-slave-f9wt"
I20260504 13:55:54.620380 9156 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260504 13:55:54.627858 9172 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 13:55:54.629029 9156 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260504 13:55:54.629387 9156 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/data,/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/wal
uuid: "806407748d564439807fe5d73db9abda"
format_stamp: "Formatted at 2026-05-04 13:55:54 on dist-test-slave-f9wt"
I20260504 13:55:54.629742 9156 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 13:55:54.714170 9156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 13:55:54.715694 9156 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 13:55:54.716156 9156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 13:55:54.789559 9156 rpc_server.cc:307] RPC server started. Bound to: 127.8.238.254:36145
I20260504 13:55:54.789628 9223 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.238.254:36145 every 8 connection(s)
I20260504 13:55:54.792122 9156 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/data/info.pb
I20260504 13:55:54.798056 9224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:55:54.799853 9147 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu as pid 9156
I20260504 13:55:54.800293 9147 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-0/wal/instance
I20260504 13:55:54.805088 9147 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
/tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.253:36337
--webserver_interface=127.8.238.253
--webserver_port=0
--builtin_ntp_servers=127.8.238.212:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.238.254:36145,127.8.238.253:36337,127.8.238.252:44781
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260504 13:55:54.809316 9224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } has no permanent_uuid. Determining permanent_uuid...
I20260504 13:55:54.837615 9224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } has no permanent_uuid. Determining permanent_uuid...
W20260504 13:55:54.840081 9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111)
W20260504 13:55:54.844328 9224 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.253:36337: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111)
I20260504 13:55:54.893898 9224 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } attempt: 1
W20260504 13:55:54.898260 9224 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.253:36337: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111)
I20260504 13:55:54.966636 9224 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } attempt: 2
W20260504 13:55:54.970851 9224 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.253:36337: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111)
I20260504 13:55:55.062224 9224 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } attempt: 3
W20260504 13:55:55.066396 9224 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.253:36337: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111)
W20260504 13:55:55.109246 9226 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260504 13:55:55.109845 9226 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260504 13:55:55.110280 9226 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260504 13:55:55.142410 9226 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260504 13:55:55.142756 9226 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260504 13:55:55.143026 9226 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260504 13:55:55.143313 9226 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260504 13:55:55.178763 9226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.238.212:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.238.254:36145,127.8.238.253:36337,127.8.238.252:44781
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.253:36337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.238.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 04 May 2026 13:43:23 UTC on e7f111948823
build id 11743
TSAN enabled
I20260504 13:55:55.180094 9226 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 13:55:55.181665 9226 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 13:55:55.192643 9235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:55:55.209836 9224 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } attempt: 4
W20260504 13:55:55.216849 9224 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.253:36337: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111)
I20260504 13:55:55.516350 9224 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } attempt: 5
W20260504 13:55:55.523430 9224 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.253:36337: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111)
I20260504 13:55:56.070966 9224 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } attempt: 6
W20260504 13:55:56.099623 9224 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.253:36337: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111)
W20260504 13:55:56.599646 9234 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ca0 after lost signal to thread 9226
W20260504 13:55:56.904826 9234 kernel_stack_watchdog.cc:198] Thread 9226 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260504 13:55:55.193706 9236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:55:56.905650 9226 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.713s user 0.665s sys 1.042s
W20260504 13:55:56.905998 9226 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.713s user 0.665s sys 1.042s
W20260504 13:55:56.907891 9238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:55:56.909842 9237 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1712 milliseconds
I20260504 13:55:56.909842 9226 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260504 13:55:56.911134 9226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260504 13:55:56.913406 9226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260504 13:55:56.914733 9226 hybrid_clock.cc:648] HybridClock initialized: now 1777902956914698 us; error 62 us; skew 500 ppm
I20260504 13:55:56.915578 9226 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 13:55:56.921267 9226 webserver.cc:492] Webserver started at http://127.8.238.253:44077/ using document root <none> and password file <none>
I20260504 13:55:56.922139 9226 fs_manager.cc:362] Metadata directory not provided
I20260504 13:55:56.922339 9226 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 13:55:56.922852 9226 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 13:55:56.926985 9226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/data/instance:
uuid: "6caedf407d13459e8fd477e8b60b7dc7"
format_stamp: "Formatted at 2026-05-04 13:55:56 on dist-test-slave-f9wt"
I20260504 13:55:56.928190 9226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/wal/instance:
uuid: "6caedf407d13459e8fd477e8b60b7dc7"
format_stamp: "Formatted at 2026-05-04 13:55:56 on dist-test-slave-f9wt"
I20260504 13:55:56.934918 9226 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260504 13:55:56.940285 9247 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 13:55:56.941257 9226 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260504 13:55:56.941579 9226 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/data,/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/wal
uuid: "6caedf407d13459e8fd477e8b60b7dc7"
format_stamp: "Formatted at 2026-05-04 13:55:56 on dist-test-slave-f9wt"
I20260504 13:55:56.941891 9226 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 13:55:57.011525 9226 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 13:55:57.012948 9226 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 13:55:57.013397 9226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 13:55:57.082118 9226 rpc_server.cc:307] RPC server started. Bound to: 127.8.238.253:36337
I20260504 13:55:57.082190 9298 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.238.253:36337 every 8 connection(s)
I20260504 13:55:57.084710 9226 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/data/info.pb
I20260504 13:55:57.090135 9299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:55:57.091440 9147 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu as pid 9226
I20260504 13:55:57.091804 9147 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/wal/instance
I20260504 13:55:57.093868 9147 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
/tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.252:44781
--webserver_interface=127.8.238.252
--webserver_port=0
--builtin_ntp_servers=127.8.238.212:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.238.254:36145,127.8.238.253:36337,127.8.238.252:44781
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260504 13:55:57.101454 9299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } has no permanent_uuid. Determining permanent_uuid...
I20260504 13:55:57.123535 9299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } has no permanent_uuid. Determining permanent_uuid...
I20260504 13:55:57.137792 9299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } has no permanent_uuid. Determining permanent_uuid...
W20260504 13:55:57.140182 9251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.252:44781: connect: Connection refused (error 111)
W20260504 13:55:57.144376 9299 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:44781: Network error: Client connection negotiation failed: client connection to 127.8.238.252:44781: connect: Connection refused (error 111)
I20260504 13:55:57.160171 9224 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } attempt: 7
I20260504 13:55:57.171850 9224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } has no permanent_uuid. Determining permanent_uuid...
W20260504 13:55:57.177578 9224 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:44781: Network error: Client connection negotiation failed: client connection to 127.8.238.252:44781: connect: Connection refused (error 111)
I20260504 13:55:57.193928 9299 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } attempt: 1
W20260504 13:55:57.197620 9299 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:44781: Network error: Client connection negotiation failed: client connection to 127.8.238.252:44781: connect: Connection refused (error 111)
I20260504 13:55:57.235948 9224 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } attempt: 1
W20260504 13:55:57.239894 9224 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:44781: Network error: Client connection negotiation failed: client connection to 127.8.238.252:44781: connect: Connection refused (error 111)
I20260504 13:55:57.266083 9299 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } attempt: 2
W20260504 13:55:57.269968 9299 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:44781: Network error: Client connection negotiation failed: client connection to 127.8.238.252:44781: connect: Connection refused (error 111)
I20260504 13:55:57.321326 9224 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } attempt: 2
W20260504 13:55:57.325400 9224 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:44781: Network error: Client connection negotiation failed: client connection to 127.8.238.252:44781: connect: Connection refused (error 111)
I20260504 13:55:57.361356 9299 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } attempt: 3
W20260504 13:55:57.365334 9299 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:44781: Network error: Client connection negotiation failed: client connection to 127.8.238.252:44781: connect: Connection refused (error 111)
W20260504 13:55:57.402644 9301 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260504 13:55:57.403246 9301 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260504 13:55:57.403658 9301 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260504 13:55:57.410761 9224 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } attempt: 3
W20260504 13:55:57.414703 9224 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:44781: Network error: Client connection negotiation failed: client connection to 127.8.238.252:44781: connect: Connection refused (error 111)
W20260504 13:55:57.436975 9301 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260504 13:55:57.437345 9301 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260504 13:55:57.437666 9301 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260504 13:55:57.437937 9301 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260504 13:55:57.474454 9301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.238.212:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.238.254:36145,127.8.238.253:36337,127.8.238.252:44781
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.252:44781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.238.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 04 May 2026 13:43:23 UTC on e7f111948823
build id 11743
TSAN enabled
I20260504 13:55:57.475937 9301 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 13:55:57.477487 9301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 13:55:57.488597 9313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:55:57.508703 9299 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } attempt: 4
W20260504 13:55:57.512418 9299 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:44781: Network error: Client connection negotiation failed: client connection to 127.8.238.252:44781: connect: Connection refused (error 111)
I20260504 13:55:57.555091 9224 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } attempt: 4
W20260504 13:55:57.561290 9224 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:44781: Network error: Client connection negotiation failed: client connection to 127.8.238.252:44781: connect: Connection refused (error 111)
I20260504 13:55:57.811815 9299 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } attempt: 5
W20260504 13:55:57.820055 9299 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:44781: Network error: Client connection negotiation failed: client connection to 127.8.238.252:44781: connect: Connection refused (error 111)
I20260504 13:55:57.844835 9224 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } attempt: 5
W20260504 13:55:57.850991 9224 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:44781: Network error: Client connection negotiation failed: client connection to 127.8.238.252:44781: connect: Connection refused (error 111)
I20260504 13:55:58.367590 9299 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } attempt: 6
W20260504 13:55:58.377444 9299 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:44781: Network error: Client connection negotiation failed: client connection to 127.8.238.252:44781: connect: Connection refused (error 111)
I20260504 13:55:58.403563 9224 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } attempt: 6
W20260504 13:55:58.413175 9224 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:44781: Network error: Client connection negotiation failed: client connection to 127.8.238.252:44781: connect: Connection refused (error 111)
W20260504 13:55:57.489835 9314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:55:58.714197 9316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:55:58.716779 9315 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1223 milliseconds
I20260504 13:55:58.716861 9301 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260504 13:55:58.718104 9301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260504 13:55:58.721119 9301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260504 13:55:58.722532 9301 hybrid_clock.cc:648] HybridClock initialized: now 1777902958722483 us; error 51 us; skew 500 ppm
I20260504 13:55:58.723402 9301 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 13:55:58.729671 9301 webserver.cc:492] Webserver started at http://127.8.238.252:35083/ using document root <none> and password file <none>
I20260504 13:55:58.730559 9301 fs_manager.cc:362] Metadata directory not provided
I20260504 13:55:58.730792 9301 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 13:55:58.731304 9301 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 13:55:58.735425 9301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/data/instance:
uuid: "ec9f4c3a43d84fc9a420dd10723c01a1"
format_stamp: "Formatted at 2026-05-04 13:55:58 on dist-test-slave-f9wt"
I20260504 13:55:58.736466 9301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/wal/instance:
uuid: "ec9f4c3a43d84fc9a420dd10723c01a1"
format_stamp: "Formatted at 2026-05-04 13:55:58 on dist-test-slave-f9wt"
I20260504 13:55:58.746146 9301 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.003s
I20260504 13:55:58.752300 9327 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 13:55:58.753297 9301 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260504 13:55:58.753584 9301 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/data,/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/wal
uuid: "ec9f4c3a43d84fc9a420dd10723c01a1"
format_stamp: "Formatted at 2026-05-04 13:55:58 on dist-test-slave-f9wt"
I20260504 13:55:58.753854 9301 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 13:55:58.810117 9301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 13:55:58.811488 9301 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 13:55:58.811905 9301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 13:55:58.880263 9301 rpc_server.cc:307] RPC server started. Bound to: 127.8.238.252:44781
I20260504 13:55:58.880326 9378 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.238.252:44781 every 8 connection(s)
I20260504 13:55:58.882856 9301 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/data/info.pb
I20260504 13:55:58.888556 9147 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu as pid 9301
I20260504 13:55:58.888792 9379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:55:58.888923 9147 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-2/wal/instance
I20260504 13:55:58.903911 9379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } has no permanent_uuid. Determining permanent_uuid...
I20260504 13:55:58.928412 9379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } has no permanent_uuid. Determining permanent_uuid...
I20260504 13:55:58.939393 9379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } has no permanent_uuid. Determining permanent_uuid...
I20260504 13:55:58.958550 9379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Bootstrap starting.
I20260504 13:55:58.964641 9379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Neither blocks nor log segments found. Creating new log.
I20260504 13:55:58.966895 9379 log.cc:826] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Log is configured to *not* fsync() on all Append() calls
I20260504 13:55:58.970984 9379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: No bootstrap required, opened a new log
I20260504 13:55:58.990655 9379 raft_consensus.cc:359] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:55:58.991318 9379 raft_consensus.cc:385] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:55:58.991537 9379 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec9f4c3a43d84fc9a420dd10723c01a1, State: Initialized, Role: FOLLOWER
I20260504 13:55:58.992163 9379 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:55:58.994307 9390 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:55:58.994868 9390 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:55:58.995625 9379 sys_catalog.cc:565] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: configured and running, proceeding with master startup.
W20260504 13:55:59.020848 9401 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:55:59.021170 9401 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260504 13:55:59.014858 9301 master_runner.cc:186] Error getting master registration for 127.8.238.254:36145: OK, instance_id { permanent_uuid: "806407748d564439807fe5d73db9abda" instance_seqno: 1777902954770265 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260504 13:55:59.034993 9301 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 13:55:59.066576 9333 tablet.cc:2404] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260504 13:55:59.390185 9390 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:55:59.390703 9390 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:55:59.393778 9390 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:55:59.437933 9299 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } attempt: 7
I20260504 13:55:59.446666 9224 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } attempt: 7
I20260504 13:55:59.471798 9299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7: Bootstrap starting.
I20260504 13:55:59.478281 9299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7: Neither blocks nor log segments found. Creating new log.
I20260504 13:55:59.480572 9299 log.cc:826] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7: Log is configured to *not* fsync() on all Append() calls
I20260504 13:55:59.491369 9299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7: No bootstrap required, opened a new log
I20260504 13:55:59.494123 9224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: Bootstrap starting.
I20260504 13:55:59.503103 9224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: Neither blocks nor log segments found. Creating new log.
I20260504 13:55:59.505414 9224 log.cc:826] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: Log is configured to *not* fsync() on all Append() calls
I20260504 13:55:59.510433 9224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: No bootstrap required, opened a new log
I20260504 13:55:59.526072 9299 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:55:59.526898 9299 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:55:59.527256 9299 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6caedf407d13459e8fd477e8b60b7dc7, State: Initialized, Role: FOLLOWER
I20260504 13:55:59.528164 9299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:55:59.532873 9299 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260504 13:55:59.533051 9274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:55:59.533910 9274 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec9f4c3a43d84fc9a420dd10723c01a1 in term 0.
I20260504 13:55:59.535300 9328 leader_election.cc:304] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7, ec9f4c3a43d84fc9a420dd10723c01a1; no voters:
I20260504 13:55:59.536101 9390 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:55:59.531741 9408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:55:59.536461 9390 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:55:59.536594 9408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:55:59.536767 9390 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:55:59.543429 9390 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:55:59.550437 9274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7"
I20260504 13:55:59.551066 9274 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:55:59.559388 9224 raft_consensus.cc:359] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:55:59.560318 9224 raft_consensus.cc:385] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:55:59.560619 9224 raft_consensus.cc:740] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 806407748d564439807fe5d73db9abda, State: Initialized, Role: FOLLOWER
I20260504 13:55:59.561453 9224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:55:59.564138 9274 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec9f4c3a43d84fc9a420dd10723c01a1 in term 1.
I20260504 13:55:59.564982 9390 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 1 election: Requested vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:55:59.565399 9328 leader_election.cc:304] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7, ec9f4c3a43d84fc9a420dd10723c01a1; no voters:
I20260504 13:55:59.566092 9390 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:55:59.565881 9415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:55:59.566593 9415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:55:59.568584 9390 raft_consensus.cc:697] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 1 LEADER]: Becoming Leader. State: Replica: ec9f4c3a43d84fc9a420dd10723c01a1, State: Running, Role: LEADER
I20260504 13:55:59.569684 9390 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:55:59.576297 9224 sys_catalog.cc:565] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: configured and running, proceeding with master startup.
I20260504 13:55:59.576521 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda"
I20260504 13:55:59.577275 9198 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:55:59.578476 9199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:55:59.582576 9417 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec9f4c3a43d84fc9a420dd10723c01a1. Latest consensus state: current_term: 1 leader_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:55:59.595640 9198 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec9f4c3a43d84fc9a420dd10723c01a1 in term 1.
I20260504 13:55:59.596149 9417 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: This master's current role is: LEADER
I20260504 13:55:59.601006 9425 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260504 13:55:59.629931 9426 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
W20260504 13:55:59.637140 9434 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:55:59.637467 9434 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260504 13:55:59.642959 9426 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260504 13:55:59.604993 9226 master_runner.cc:186] Error getting master registration for 127.8.238.254:36145: OK, instance_id { permanent_uuid: "806407748d564439807fe5d73db9abda" instance_seqno: 1777902954770265 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260504 13:55:59.652887 9226 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:55:59.667604 9274 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Refusing update from remote peer ec9f4c3a43d84fc9a420dd10723c01a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 13:55:59.669283 9417 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:55:59.676153 9198 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Refusing update from remote peer ec9f4c3a43d84fc9a420dd10723c01a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 13:55:59.677901 9417 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:55:59.706027 9436 mvcc.cc:204] Tried to move back new op lower bound from 7282290522778861568 to 7282290522408325120. Current Snapshot: MvccSnapshot[applied={T|T < 7282290522778861568}]
I20260504 13:55:59.710765 9408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec9f4c3a43d84fc9a420dd10723c01a1. Latest consensus state: current_term: 1 leader_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:55:59.711887 9408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:55:59.715842 9390 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:55:59.716535 9390 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: This master's current role is: LEADER
I20260504 13:55:59.718217 9426 catalog_manager.cc:1357] Generated new cluster ID: 302d3e241578494980ac3509cf5306fb
I20260504 13:55:59.718600 9426 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260504 13:55:59.724498 9415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec9f4c3a43d84fc9a420dd10723c01a1. Latest consensus state: current_term: 1 leader_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:55:59.725162 9415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: This master's current role is: FOLLOWER
W20260504 13:55:59.726060 9178 tablet.cc:2404] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260504 13:55:59.727649 9417 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:55:59.728327 9417 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: This master's current role is: LEADER
I20260504 13:55:59.732470 9438 mvcc.cc:204] Tried to move back new op lower bound from 7282290522778861568 to 7282290522408325120. Current Snapshot: MvccSnapshot[applied={T|T < 7282290522778861568}]
I20260504 13:55:59.741309 9415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:55:59.742381 9415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:55:59.757261 9408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:55:59.758041 9408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:55:59.779011 9426 catalog_manager.cc:1380] Generated new certificate authority record
I20260504 13:55:59.780954 9426 catalog_manager.cc:1514] Loading token signing keys...
I20260504 13:55:59.812589 9426 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Generated new TSK 0
I20260504 13:55:59.813642 9426 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260504 13:55:59.863039 9147 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
/tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.238.193:0
--local_ip_for_outbound_sockets=127.8.238.193
--webserver_interface=127.8.238.193
--webserver_port=0
--tserver_master_addrs=127.8.238.254:36145,127.8.238.253:36337,127.8.238.252:44781
--builtin_ntp_servers=127.8.238.212:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20260504 13:56:00.316659 9445 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260504 13:56:00.317306 9445 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260504 13:56:00.318203 9445 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260504 13:56:00.376519 9445 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260504 13:56:00.378005 9445 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.238.193
I20260504 13:56:00.436499 9445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.238.212:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.238.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.238.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.8.238.254:36145,127.8.238.253:36337,127.8.238.252:44781
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.238.193
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 04 May 2026 13:43:23 UTC on e7f111948823
build id 11743
TSAN enabled
I20260504 13:56:00.437851 9445 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 13:56:00.439523 9445 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 13:56:00.452946 9451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:56:00.627238 9425 catalog_manager.cc:1269] Loaded cluster ID: 302d3e241578494980ac3509cf5306fb
I20260504 13:56:00.629614 9425 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7: loading cluster ID for follower catalog manager: success
I20260504 13:56:00.643023 9434 catalog_manager.cc:1269] Loaded cluster ID: 302d3e241578494980ac3509cf5306fb
I20260504 13:56:00.643354 9434 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: loading cluster ID for follower catalog manager: success
I20260504 13:56:00.652846 9434 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: acquiring CA information for follower catalog manager: success
I20260504 13:56:00.658785 9434 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260504 13:56:00.667438 9425 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7: acquiring CA information for follower catalog manager: success
I20260504 13:56:00.688086 9425 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260504 13:56:00.739988 9458 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:00.740614 9458 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:00.744234 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:00.755946 9458 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:00.756841 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:00.758569 9251 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:00.759666 9458 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:01.176023 9459 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:01.187443 9459 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:01.191519 9273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:01.195226 9459 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:01.195525 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:01.198102 9176 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 806407748d564439807fe5d73db9abda; no voters: 6caedf407d13459e8fd477e8b60b7dc7, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:01.199239 9459 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:01.289345 9460 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:01.289839 9460 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:01.301239 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:01.304224 9460 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:01.304806 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:01.307144 9251 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:01.308214 9460 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:01.751379 9461 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:01.752405 9461 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:01.755438 9273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:01.760484 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:01.761898 9176 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 806407748d564439807fe5d73db9abda; no voters: 6caedf407d13459e8fd477e8b60b7dc7, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:01.767539 9461 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:01.768502 9461 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:01.886060 9463 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:01.892987 9463 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:01.896625 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:01.900403 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:01.902212 9251 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:01.904004 9463 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:01.905635 9464 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:56:01.853461 9450 debug-util.cc:398] Leaking SignalData structure 0x7b0800028ca0 after lost signal to thread 9445
W20260504 13:56:02.744386 9450 kernel_stack_watchdog.cc:198] Thread 9445 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 396ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260504 13:56:00.454627 9452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:56:02.747481 9453 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2290 milliseconds
W20260504 13:56:02.748456 9454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:56:02.748991 9445 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.292s user 0.612s sys 1.007s
W20260504 13:56:02.749363 9445 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.294s user 0.612s sys 1.007s
I20260504 13:56:02.749660 9445 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260504 13:56:02.753532 9445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260504 13:56:02.759670 9445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260504 13:56:02.762244 9445 hybrid_clock.cc:648] HybridClock initialized: now 1777902962762202 us; error 43 us; skew 500 ppm
I20260504 13:56:02.763435 9445 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 13:56:02.771867 9467 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:02.772359 9467 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:02.774346 9467 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:02.775038 9273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:02.777446 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:02.779038 9176 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 806407748d564439807fe5d73db9abda; no voters: 6caedf407d13459e8fd477e8b60b7dc7, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:02.779923 9467 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:02.781879 9445 webserver.cc:492] Webserver started at http://127.8.238.193:37687/ using document root <none> and password file <none>
I20260504 13:56:02.783100 9445 fs_manager.cc:362] Metadata directory not provided
I20260504 13:56:02.783430 9445 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 13:56:02.784103 9445 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 13:56:02.790242 9445 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/data/instance:
uuid: "7b79fe7dd8934dd996393a2d96eabf00"
format_stamp: "Formatted at 2026-05-04 13:56:02 on dist-test-slave-f9wt"
I20260504 13:56:02.791738 9445 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/instance:
uuid: "7b79fe7dd8934dd996393a2d96eabf00"
format_stamp: "Formatted at 2026-05-04 13:56:02 on dist-test-slave-f9wt"
I20260504 13:56:02.806263 9445 fs_manager.cc:696] Time spent creating directory manager: real 0.014s user 0.004s sys 0.008s
I20260504 13:56:02.828426 9470 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 13:56:02.830350 9445 fs_manager.cc:730] Time spent opening block manager: real 0.020s user 0.010s sys 0.011s
I20260504 13:56:02.830739 9445 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal
uuid: "7b79fe7dd8934dd996393a2d96eabf00"
format_stamp: "Formatted at 2026-05-04 13:56:02 on dist-test-slave-f9wt"
I20260504 13:56:02.831151 9445 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 13:56:02.958715 9445 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 13:56:02.960650 9445 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 13:56:02.961227 9445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 13:56:02.964767 9445 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 13:56:02.970469 9445 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 13:56:02.970738 9445 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 13:56:02.971025 9445 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 13:56:02.971292 9445 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260504 13:56:03.182592 9445 rpc_server.cc:307] RPC server started. Bound to: 127.8.238.193:33117
I20260504 13:56:03.182718 9584 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.238.193:33117 every 8 connection(s)
I20260504 13:56:03.186199 9445 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/data/info.pb
I20260504 13:56:03.188562 9147 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu as pid 9445
I20260504 13:56:03.189214 9147 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/instance
I20260504 13:56:03.204931 9147 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
/tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.238.194:0
--local_ip_for_outbound_sockets=127.8.238.194
--webserver_interface=127.8.238.194
--webserver_port=0
--tserver_master_addrs=127.8.238.254:36145,127.8.238.253:36337,127.8.238.252:44781
--builtin_ntp_servers=127.8.238.212:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260504 13:56:03.267037 9585 heartbeater.cc:344] Connected to a master server at 127.8.238.252:44781
I20260504 13:56:03.267678 9585 heartbeater.cc:461] Registering TS with master...
I20260504 13:56:03.269243 9585 heartbeater.cc:507] Master 127.8.238.252:44781 requested a full tablet report, sending...
I20260504 13:56:03.272617 9344 ts_manager.cc:194] Registered new tserver with Master: 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:03.275235 9344 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.238.193:38691
I20260504 13:56:03.286609 9587 heartbeater.cc:344] Connected to a master server at 127.8.238.253:36337
I20260504 13:56:03.286976 9587 heartbeater.cc:461] Registering TS with master...
I20260504 13:56:03.287636 9587 heartbeater.cc:507] Master 127.8.238.253:36337 requested a full tablet report, sending...
I20260504 13:56:03.290673 9263 ts_manager.cc:194] Registered new tserver with Master: 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:03.307637 9588 heartbeater.cc:344] Connected to a master server at 127.8.238.254:36145
I20260504 13:56:03.308022 9588 heartbeater.cc:461] Registering TS with master...
I20260504 13:56:03.308666 9588 heartbeater.cc:507] Master 127.8.238.254:36145 requested a full tablet report, sending...
I20260504 13:56:03.312310 9188 ts_manager.cc:194] Registered new tserver with Master: 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:03.367722 9596 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:03.368865 9596 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:03.373189 9273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:03.376816 9596 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:03.376983 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:03.378545 9176 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 806407748d564439807fe5d73db9abda; no voters: 6caedf407d13459e8fd477e8b60b7dc7, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:03.380064 9596 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:56:03.609059 9586 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260504 13:56:03.609575 9586 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260504 13:56:03.610145 9586 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260504 13:56:03.642367 9586 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260504 13:56:03.643368 9586 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.238.194
I20260504 13:56:03.681725 9586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.238.212:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.238.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.238.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.8.238.254:36145,127.8.238.253:36337,127.8.238.252:44781
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.238.194
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 04 May 2026 13:43:23 UTC on e7f111948823
build id 11743
TSAN enabled
I20260504 13:56:03.683040 9586 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 13:56:03.684656 9586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 13:56:03.697211 9602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:56:03.779521 9585 heartbeater.cc:499] Master 127.8.238.252:44781 was elected leader, sending a full tablet report...
I20260504 13:56:04.340915 9608 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:04.341545 9608 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:04.344401 9608 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:04.344204 9273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:04.344550 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:04.346475 9173 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 806407748d564439807fe5d73db9abda; no voters: 6caedf407d13459e8fd477e8b60b7dc7, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:04.347482 9608 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:04.942340 9609 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:04.944589 9609 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:04.948875 9273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:04.951212 9609 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:04.951073 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:04.952739 9176 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 806407748d564439807fe5d73db9abda; no voters: 6caedf407d13459e8fd477e8b60b7dc7, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:04.953967 9609 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:05.063622 9610 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:05.067409 9610 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:05.071158 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:05.077235 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:05.079241 9251 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:05.084245 9610 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:05.085039 9610 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:56:05.104899 9581 debug-util.cc:398] Leaking SignalData structure 0x7b08000ad060 after lost signal to thread 9446
W20260504 13:56:05.105988 9581 debug-util.cc:398] Leaking SignalData structure 0x7b08000adf40 after lost signal to thread 9584
W20260504 13:56:05.098737 9601 debug-util.cc:398] Leaking SignalData structure 0x7b0800028ca0 after lost signal to thread 9586
W20260504 13:56:05.503511 9601 kernel_stack_watchdog.cc:198] Thread 9586 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 397ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260504 13:56:03.698516 9603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:56:05.509208 9586 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.810s user 0.003s sys 0.001s
W20260504 13:56:05.509804 9586 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.810s user 0.004s sys 0.001s
W20260504 13:56:05.511271 9605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:56:05.512576 9604 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1813 milliseconds
I20260504 13:56:05.512625 9586 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260504 13:56:05.514118 9586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260504 13:56:05.516736 9586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260504 13:56:05.518193 9586 hybrid_clock.cc:648] HybridClock initialized: now 1777902965518142 us; error 51 us; skew 500 ppm
I20260504 13:56:05.518976 9586 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 13:56:05.525804 9586 webserver.cc:492] Webserver started at http://127.8.238.194:39305/ using document root <none> and password file <none>
I20260504 13:56:05.527096 9586 fs_manager.cc:362] Metadata directory not provided
I20260504 13:56:05.527397 9586 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 13:56:05.528012 9586 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 13:56:05.534689 9586 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/data/instance:
uuid: "5beceb6317444247b31b44a6581707c6"
format_stamp: "Formatted at 2026-05-04 13:56:05 on dist-test-slave-f9wt"
I20260504 13:56:05.536348 9586 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/instance:
uuid: "5beceb6317444247b31b44a6581707c6"
format_stamp: "Formatted at 2026-05-04 13:56:05 on dist-test-slave-f9wt"
I20260504 13:56:05.545737 9586 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260504 13:56:05.552011 9617 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 13:56:05.553188 9586 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260504 13:56:05.553510 9586 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal
uuid: "5beceb6317444247b31b44a6581707c6"
format_stamp: "Formatted at 2026-05-04 13:56:05 on dist-test-slave-f9wt"
I20260504 13:56:05.553867 9586 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 13:56:05.606935 9622 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:05.607627 9622 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:05.609498 9622 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:05.610569 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:05.610667 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:05.612459 9251 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:05.613485 9622 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:05.615510 9586 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 13:56:05.617108 9586 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 13:56:05.617530 9586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 13:56:05.620514 9586 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 13:56:05.624677 9586 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 13:56:05.624902 9586 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 13:56:05.625169 9586 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 13:56:05.625352 9586 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260504 13:56:05.811898 9586 rpc_server.cc:307] RPC server started. Bound to: 127.8.238.194:40369
I20260504 13:56:05.812005 9730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.238.194:40369 every 8 connection(s)
I20260504 13:56:05.814608 9586 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/data/info.pb
I20260504 13:56:05.817718 9147 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu as pid 9586
I20260504 13:56:05.818313 9147 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/instance
I20260504 13:56:05.834585 9147 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
/tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.238.195:0
--local_ip_for_outbound_sockets=127.8.238.195
--webserver_interface=127.8.238.195
--webserver_port=0
--tserver_master_addrs=127.8.238.254:36145,127.8.238.253:36337,127.8.238.252:44781
--builtin_ntp_servers=127.8.238.212:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260504 13:56:05.925535 9736 heartbeater.cc:344] Connected to a master server at 127.8.238.254:36145
I20260504 13:56:05.926084 9736 heartbeater.cc:461] Registering TS with master...
I20260504 13:56:05.927384 9736 heartbeater.cc:507] Master 127.8.238.254:36145 requested a full tablet report, sending...
I20260504 13:56:05.930306 9188 ts_manager.cc:194] Registered new tserver with Master: 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:05.932248 9731 heartbeater.cc:344] Connected to a master server at 127.8.238.252:44781
I20260504 13:56:05.932608 9731 heartbeater.cc:461] Registering TS with master...
I20260504 13:56:05.933521 9731 heartbeater.cc:507] Master 127.8.238.252:44781 requested a full tablet report, sending...
I20260504 13:56:05.936818 9344 ts_manager.cc:194] Registered new tserver with Master: 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:05.938531 9344 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.238.194:38543
I20260504 13:56:05.934458 9732 heartbeater.cc:344] Connected to a master server at 127.8.238.253:36337
I20260504 13:56:05.942535 9732 heartbeater.cc:461] Registering TS with master...
I20260504 13:56:05.943260 9732 heartbeater.cc:507] Master 127.8.238.253:36337 requested a full tablet report, sending...
I20260504 13:56:05.945770 9263 ts_manager.cc:194] Registered new tserver with Master: 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:06.002951 9742 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:06.003494 9742 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:06.005570 9742 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:06.006543 9273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:06.007129 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:06.008635 9176 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 806407748d564439807fe5d73db9abda; no voters: 6caedf407d13459e8fd477e8b60b7dc7, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:06.009433 9742 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:06.133445 9743 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:06.134136 9743 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:06.137091 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:06.140308 9743 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:06.140491 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:06.142045 9251 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:06.143201 9743 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:56:06.228181 9735 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260504 13:56:06.228693 9735 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260504 13:56:06.229250 9735 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260504 13:56:06.261389 9735 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260504 13:56:06.262346 9735 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.238.195
I20260504 13:56:06.300334 9735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.238.212:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.238.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.238.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.8.238.254:36145,127.8.238.253:36337,127.8.238.252:44781
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.238.195
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 04 May 2026 13:43:23 UTC on e7f111948823
build id 11743
TSAN enabled
I20260504 13:56:06.301677 9735 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 13:56:06.303267 9735 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 13:56:06.317091 9749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:56:06.442561 9731 heartbeater.cc:499] Master 127.8.238.252:44781 was elected leader, sending a full tablet report...
I20260504 13:56:06.577673 9755 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:06.585268 9755 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:06.595606 9273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:06.599573 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:06.601383 9176 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 806407748d564439807fe5d73db9abda; no voters: 6caedf407d13459e8fd477e8b60b7dc7, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:06.607111 9755 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:06.611449 9755 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:07.257915 9757 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:07.258811 9757 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:07.262481 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:07.275043 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:07.277340 9251 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:07.279848 9757 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:07.281011 9757 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:07.491073 9760 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:07.493206 9760 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:07.498091 9273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:07.498827 9760 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:07.500957 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:07.503974 9176 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 806407748d564439807fe5d73db9abda; no voters: 6caedf407d13459e8fd477e8b60b7dc7, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:07.504904 9760 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:07.859899 9761 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:07.860455 9761 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:07.862427 9761 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:07.865633 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:07.869127 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:07.871558 9251 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:07.872684 9761 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:56:07.718950 9748 debug-util.cc:398] Leaking SignalData structure 0x7b0800028ca0 after lost signal to thread 9735
W20260504 13:56:08.304255 9748 kernel_stack_watchdog.cc:198] Thread 9735 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 397ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260504 13:56:06.318310 9750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:56:08.305927 9735 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.987s user 0.000s sys 0.002s
W20260504 13:56:08.306273 9735 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.987s user 0.000s sys 0.002s
W20260504 13:56:08.308650 9752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:56:08.311502 9751 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1992 milliseconds
I20260504 13:56:08.311681 9735 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260504 13:56:08.312791 9735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260504 13:56:08.314731 9735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260504 13:56:08.316104 9735 hybrid_clock.cc:648] HybridClock initialized: now 1777902968316093 us; error 56 us; skew 500 ppm
I20260504 13:56:08.316877 9735 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 13:56:08.322561 9735 webserver.cc:492] Webserver started at http://127.8.238.195:45285/ using document root <none> and password file <none>
I20260504 13:56:08.323499 9735 fs_manager.cc:362] Metadata directory not provided
I20260504 13:56:08.323726 9735 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 13:56:08.324153 9735 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 13:56:08.328449 9735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/data/instance:
uuid: "d4051d2bf0bf410eb8acd107560df888"
format_stamp: "Formatted at 2026-05-04 13:56:08 on dist-test-slave-f9wt"
I20260504 13:56:08.329941 9735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/instance:
uuid: "d4051d2bf0bf410eb8acd107560df888"
format_stamp: "Formatted at 2026-05-04 13:56:08 on dist-test-slave-f9wt"
I20260504 13:56:08.337291 9735 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260504 13:56:08.342837 9766 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 13:56:08.343822 9735 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260504 13:56:08.344139 9735 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal
uuid: "d4051d2bf0bf410eb8acd107560df888"
format_stamp: "Formatted at 2026-05-04 13:56:08 on dist-test-slave-f9wt"
I20260504 13:56:08.344491 9735 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 13:56:08.393040 9735 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 13:56:08.394397 9735 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 13:56:08.394819 9735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 13:56:08.397428 9735 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 13:56:08.401465 9735 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 13:56:08.401690 9735 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 13:56:08.401940 9735 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 13:56:08.402122 9735 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260504 13:56:08.538100 9735 rpc_server.cc:307] RPC server started. Bound to: 127.8.238.195:34563
I20260504 13:56:08.538196 9878 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.238.195:34563 every 8 connection(s)
I20260504 13:56:08.540712 9735 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/data/info.pb
I20260504 13:56:08.549678 9147 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu as pid 9735
I20260504 13:56:08.550283 9147 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/instance
I20260504 13:56:08.590099 9880 heartbeater.cc:344] Connected to a master server at 127.8.238.253:36337
I20260504 13:56:08.590745 9880 heartbeater.cc:461] Registering TS with master...
I20260504 13:56:08.592011 9880 heartbeater.cc:507] Master 127.8.238.253:36337 requested a full tablet report, sending...
I20260504 13:56:08.592450 9879 heartbeater.cc:344] Connected to a master server at 127.8.238.252:44781
I20260504 13:56:08.592767 9879 heartbeater.cc:461] Registering TS with master...
I20260504 13:56:08.593652 9879 heartbeater.cc:507] Master 127.8.238.252:44781 requested a full tablet report, sending...
I20260504 13:56:08.594946 9263 ts_manager.cc:194] Registered new tserver with Master: d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:08.596613 9344 ts_manager.cc:194] Registered new tserver with Master: d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:08.597368 9881 heartbeater.cc:344] Connected to a master server at 127.8.238.254:36145
I20260504 13:56:08.597651 9881 heartbeater.cc:461] Registering TS with master...
I20260504 13:56:08.598335 9344 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.238.195:55533
I20260504 13:56:08.598451 9881 heartbeater.cc:507] Master 127.8.238.254:36145 requested a full tablet report, sending...
I20260504 13:56:08.601053 9188 ts_manager.cc:194] Registered new tserver with Master: d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:08.603073 9147 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260504 13:56:08.658903 9344 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_c1ca8ba1df7249559400d847556e8e4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:08.660019 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_4a21879694584114be26d99c3a3eb97b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:08.660229 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_77ae0c75b99942e39937f67729a06929"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:08.662768 9344 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1ca8ba1df7249559400d847556e8e4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:08.662400 9343 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_20ed3c2f67ff42d3baa15765e662d2b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:08.664541 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77ae0c75b99942e39937f67729a06929 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:08.664728 9343 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20ed3c2f67ff42d3baa15765e662d2b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:08.667834 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a21879694584114be26d99c3a3eb97b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:08.708127 9147 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu with pid 9226
I20260504 13:56:08.725839 9344 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_c1ca8ba1df7249559400d847556e8e4a" } new_table_name: "default.Table_5e708910c26b41f6ae1ed5871224320f" modify_external_catalogs: true
I20260504 13:56:08.748278 9341 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_77ae0c75b99942e39937f67729a06929" } new_table_name: "default.Table_83c39f83c18b4f6fb7bbbae7136b1c30" modify_external_catalogs: true
I20260504 13:56:08.751610 9342 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_4a21879694584114be26d99c3a3eb97b" } modify_external_catalogs: true
I20260504 13:56:08.752336 9342 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_4a21879694584114be26d99c3a3eb97b" } modify_external_catalogs: true
I20260504 13:56:08.758793 9343 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_20ed3c2f67ff42d3baa15765e662d2b0" } modify_external_catalogs: true
I20260504 13:56:08.759452 9343 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_20ed3c2f67ff42d3baa15765e662d2b0" } modify_external_catalogs: true
W20260504 13:56:08.762594 9328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.238.253:36337 (error 108)
W20260504 13:56:08.784629 9344 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6a6b276cfa14a85a5324728a12b4609 consensus state has no leader
W20260504 13:56:08.821228 9341 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6182f0cd5474ca69d1ccb4963dd1144 has no consensus state
I20260504 13:56:08.822346 9342 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet ee05f195063d4858ab6b698b095504df
W20260504 13:56:08.827313 9328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 64859.9us
W20260504 13:56:08.832008 9329 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet c6a6b276cfa14a85a5324728a12b4609 (table default.Table_5e708910c26b41f6ae1ed5871224320f [id=15c00efe5d7e4f2586f9f0dab975346f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c6a6b276cfa14a85a5324728a12b4609 consensus state has no leader
I20260504 13:56:08.833608 9814 tablet_service.cc:1511] Processing CreateTablet for tablet c6a6b276cfa14a85a5324728a12b4609 (DEFAULT_TABLE table=default.Table_c1ca8ba1df7249559400d847556e8e4a [id=15c00efe5d7e4f2586f9f0dab975346f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:08.839200 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a6b276cfa14a85a5324728a12b4609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:08.839221 9666 tablet_service.cc:1511] Processing CreateTablet for tablet c6a6b276cfa14a85a5324728a12b4609 (DEFAULT_TABLE table=default.Table_c1ca8ba1df7249559400d847556e8e4a [id=15c00efe5d7e4f2586f9f0dab975346f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:08.841075 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a6b276cfa14a85a5324728a12b4609. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:08.843142 9587 heartbeater.cc:646] Failed to heartbeat to 127.8.238.253:36337 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111)
I20260504 13:56:08.846438 9343 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 89d13cef95134578a9b83f09cabef704
W20260504 13:56:08.869046 9328 consensus_peers.cc:597] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 -> Peer 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337): Couldn't send request to peer 6caedf407d13459e8fd477e8b60b7dc7. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:08.873652 9813 tablet_service.cc:1511] Processing CreateTablet for tablet b6182f0cd5474ca69d1ccb4963dd1144 (DEFAULT_TABLE table=default.Table_83c39f83c18b4f6fb7bbbae7136b1c30 [id=fc0aa7190c444823b365c3399bba1aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:08.873633 9665 tablet_service.cc:1511] Processing CreateTablet for tablet b6182f0cd5474ca69d1ccb4963dd1144 (DEFAULT_TABLE table=default.Table_83c39f83c18b4f6fb7bbbae7136b1c30 [id=fc0aa7190c444823b365c3399bba1aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:08.874852 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6182f0cd5474ca69d1ccb4963dd1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:08.876242 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6182f0cd5474ca69d1ccb4963dd1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:08.876317 9519 tablet_service.cc:1511] Processing CreateTablet for tablet b6182f0cd5474ca69d1ccb4963dd1144 (DEFAULT_TABLE table=default.Table_83c39f83c18b4f6fb7bbbae7136b1c30 [id=fc0aa7190c444823b365c3399bba1aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:08.878412 9519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6182f0cd5474ca69d1ccb4963dd1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:08.887327 9520 tablet_service.cc:1511] Processing CreateTablet for tablet c6a6b276cfa14a85a5324728a12b4609 (DEFAULT_TABLE table=default.Table_c1ca8ba1df7249559400d847556e8e4a [id=15c00efe5d7e4f2586f9f0dab975346f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:08.888674 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a6b276cfa14a85a5324728a12b4609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:08.901044 9931 tablet_bootstrap.cc:492] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:08.910955 9931 tablet_bootstrap.cc:654] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:08.913321 9931 log.cc:826] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888: Log is configured to *not* fsync() on all Append() calls
I20260504 13:56:08.926091 9343 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_cc1e243c7c1941fd9643eff155e02f4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:08.927922 9343 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc1e243c7c1941fd9643eff155e02f4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:08.935667 9932 tablet_bootstrap.cc:492] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:08.949592 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_74913fc2d4ca4b7abef7ec95e595cea0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:08.952201 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_9242dce1382d42b2b059a7d4b480b5dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:08.954162 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74913fc2d4ca4b7abef7ec95e595cea0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:08.960333 9344 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_359d50cb39d6400d96b891c6866aa520"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:08.960788 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9242dce1382d42b2b059a7d4b480b5dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:08.961994 9344 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_359d50cb39d6400d96b891c6866aa520 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:08.985703 9932 tablet_bootstrap.cc:654] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:08.993273 9934 tablet_bootstrap.cc:492] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:08.994092 9732 heartbeater.cc:646] Failed to heartbeat to 127.8.238.253:36337 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111)
I20260504 13:56:09.003232 9932 log.cc:826] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6: Log is configured to *not* fsync() on all Append() calls
I20260504 13:56:09.010020 9934 tablet_bootstrap.cc:654] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:09.018357 9520 tablet_service.cc:1511] Processing CreateTablet for tablet 357407221df147df95940c492a38f3e8 (DEFAULT_TABLE table=default.Table_cc1e243c7c1941fd9643eff155e02f4f [id=4ae0e62b5bad4723a28421c4aba76e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.019500 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 357407221df147df95940c492a38f3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.020139 9665 tablet_service.cc:1511] Processing CreateTablet for tablet 357407221df147df95940c492a38f3e8 (DEFAULT_TABLE table=default.Table_cc1e243c7c1941fd9643eff155e02f4f [id=4ae0e62b5bad4723a28421c4aba76e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.021385 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 357407221df147df95940c492a38f3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.024873 9814 tablet_service.cc:1511] Processing CreateTablet for tablet 357407221df147df95940c492a38f3e8 (DEFAULT_TABLE table=default.Table_cc1e243c7c1941fd9643eff155e02f4f [id=4ae0e62b5bad4723a28421c4aba76e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.025987 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 357407221df147df95940c492a38f3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.034142 9931 tablet_bootstrap.cc:492] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:09.034754 9931 ts_tablet_manager.cc:1403] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.134s user 0.011s sys 0.010s
I20260504 13:56:09.054147 9342 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_5e708910c26b41f6ae1ed5871224320f" } new_table_name: "default.Table_52c096e5c7f945ba9c54cb06e0d7833e" modify_external_catalogs: true
I20260504 13:56:09.057817 9932 tablet_bootstrap.cc:492] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:09.058393 9932 ts_tablet_manager.cc:1403] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.128s user 0.011s sys 0.009s
I20260504 13:56:09.064771 9934 log.cc:826] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00: Log is configured to *not* fsync() on all Append() calls
I20260504 13:56:09.073766 9341 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_359d50cb39d6400d96b891c6866aa520" } new_table_name: "default.Table_b4f9cf9c2ee740929d78f0d29f09ba92" modify_external_catalogs: true
I20260504 13:56:09.075768 9931 raft_consensus.cc:359] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:09.076757 9931 raft_consensus.cc:385] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:09.077121 9931 raft_consensus.cc:740] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:09.078583 9931 consensus_queue.cc:260] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:09.080235 9344 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_74913fc2d4ca4b7abef7ec95e595cea0" } modify_external_catalogs: true
I20260504 13:56:09.080873 9344 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_74913fc2d4ca4b7abef7ec95e595cea0" } modify_external_catalogs: true
I20260504 13:56:09.084632 9343 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_83c39f83c18b4f6fb7bbbae7136b1c30" } modify_external_catalogs: true
I20260504 13:56:09.085145 9343 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_83c39f83c18b4f6fb7bbbae7136b1c30" } modify_external_catalogs: true
I20260504 13:56:09.092113 9934 tablet_bootstrap.cc:492] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:09.092743 9934 ts_tablet_manager.cc:1403] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.100s user 0.017s sys 0.000s
I20260504 13:56:09.098054 9520 tablet_service.cc:1511] Processing CreateTablet for tablet 2951cbd2df4f402eabeef323d121ce1b (DEFAULT_TABLE table=default.Table_359d50cb39d6400d96b891c6866aa520 [id=46ac1f0baae0492984e87927e4905343]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.099133 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2951cbd2df4f402eabeef323d121ce1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.102048 9879 heartbeater.cc:499] Master 127.8.238.252:44781 was elected leader, sending a full tablet report...
I20260504 13:56:09.103927 9814 tablet_service.cc:1511] Processing CreateTablet for tablet 2951cbd2df4f402eabeef323d121ce1b (DEFAULT_TABLE table=default.Table_359d50cb39d6400d96b891c6866aa520 [id=46ac1f0baae0492984e87927e4905343]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.104957 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2951cbd2df4f402eabeef323d121ce1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.119417 9344 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet ab451e93a2fc44758c4096e58f77a238
W20260504 13:56:09.122957 9341 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2951cbd2df4f402eabeef323d121ce1b consensus state has no leader
I20260504 13:56:09.130148 9665 tablet_service.cc:1511] Processing CreateTablet for tablet 2951cbd2df4f402eabeef323d121ce1b (DEFAULT_TABLE table=default.Table_359d50cb39d6400d96b891c6866aa520 [id=46ac1f0baae0492984e87927e4905343]), partition=RANGE (key) PARTITION UNBOUNDED
W20260504 13:56:09.132026 9342 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6a6b276cfa14a85a5324728a12b4609 consensus state has no leader
I20260504 13:56:09.139096 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2951cbd2df4f402eabeef323d121ce1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.150945 9932 raft_consensus.cc:359] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:09.152251 9932 raft_consensus.cc:385] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:09.152673 9932 raft_consensus.cc:740] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:09.153751 9932 consensus_queue.cc:260] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:09.161051 9932 ts_tablet_manager.cc:1434] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.102s user 0.032s sys 0.003s
I20260504 13:56:09.157218 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_9fb74f3a10f641028449869ce8600210"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:09.163419 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9fb74f3a10f641028449869ce8600210 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:09.167301 9932 tablet_bootstrap.cc:492] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:09.167649 9739 tablet.cc:2404] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260504 13:56:09.171216 9931 ts_tablet_manager.cc:1434] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.136s user 0.036s sys 0.002s
I20260504 13:56:09.173451 9931 tablet_bootstrap.cc:492] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:09.187103 9932 tablet_bootstrap.cc:654] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:09.187871 9931 tablet_bootstrap.cc:654] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:09.195545 9344 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_5d8e09780e014609b010c3b0b49726eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:09.197254 9344 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d8e09780e014609b010c3b0b49726eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:09.211550 9932 tablet_bootstrap.cc:492] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:09.211900 9931 tablet_bootstrap.cc:492] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:09.212014 9932 ts_tablet_manager.cc:1403] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.011s
I20260504 13:56:09.212364 9931 ts_tablet_manager.cc:1403] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.006s
I20260504 13:56:09.214823 9932 raft_consensus.cc:359] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:09.215502 9932 raft_consensus.cc:385] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:09.215795 9932 raft_consensus.cc:740] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:09.216446 9932 consensus_queue.cc:260] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:09.217345 9340 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet c6a6b276cfa14a85a5324728a12b4609 (table default.Table_52c096e5c7f945ba9c54cb06e0d7833e [id=15c00efe5d7e4f2586f9f0dab975346f]). Expected version 2 got 0
I20260504 13:56:09.217294 9520 tablet_service.cc:1511] Processing CreateTablet for tablet 5625163f6add4a20b1828905d90b5486 (DEFAULT_TABLE table=default.Table_9242dce1382d42b2b059a7d4b480b5dd [id=7b3ef250728944d9998ef2df032f8819]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.218309 9343 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet b6182f0cd5474ca69d1ccb4963dd1144
I20260504 13:56:09.218753 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5625163f6add4a20b1828905d90b5486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.221323 9666 tablet_service.cc:1558] Processing DeleteTablet for tablet b6182f0cd5474ca69d1ccb4963dd1144 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:09 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:09.226737 9932 ts_tablet_manager.cc:1434] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.014s user 0.004s sys 0.001s
I20260504 13:56:09.226729 9519 tablet_service.cc:1558] Processing DeleteTablet for tablet b6182f0cd5474ca69d1ccb4963dd1144 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:09 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:09.229126 9931 raft_consensus.cc:359] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:09.229803 9931 raft_consensus.cc:385] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:09.230101 9931 raft_consensus.cc:740] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:09.234529 9931 consensus_queue.cc:260] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:09.236450 9932 tablet_bootstrap.cc:492] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:09.236053 9934 raft_consensus.cc:359] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:09.240561 9931 ts_tablet_manager.cc:1434] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.028s user 0.000s sys 0.005s
I20260504 13:56:09.241798 9934 raft_consensus.cc:385] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:09.244859 9931 tablet_bootstrap.cc:492] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:09.245178 9934 raft_consensus.cc:740] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:09.246448 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_ba0d3ea2e915469cbb3737d5d7d1a56d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:09.247258 9932 tablet_bootstrap.cc:654] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:09.248219 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba0d3ea2e915469cbb3737d5d7d1a56d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:09.251543 9343 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet c6a6b276cfa14a85a5324728a12b4609 (table default.Table_52c096e5c7f945ba9c54cb06e0d7833e [id=15c00efe5d7e4f2586f9f0dab975346f]). Expected version 2 got 0
I20260504 13:56:09.252887 9934 consensus_queue.cc:260] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:09.249046 9913 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:09.261133 9913 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: This master's current role is: LEADER
I20260504 13:56:09.263964 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_358b445896d0436d9eebae53ffa9691c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:09.265923 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_358b445896d0436d9eebae53ffa9691c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:09.272279 9664 tablet_service.cc:1511] Processing CreateTablet for tablet 5625163f6add4a20b1828905d90b5486 (DEFAULT_TABLE table=default.Table_9242dce1382d42b2b059a7d4b480b5dd [id=7b3ef250728944d9998ef2df032f8819]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.274066 9664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5625163f6add4a20b1828905d90b5486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.275233 9931 tablet_bootstrap.cc:654] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:09.260669 9342 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet c6a6b276cfa14a85a5324728a12b4609 (table default.Table_52c096e5c7f945ba9c54cb06e0d7833e [id=15c00efe5d7e4f2586f9f0dab975346f]). Expected version 2 got 0
I20260504 13:56:09.282433 9343 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet c6a6b276cfa14a85a5324728a12b4609 (table default.Table_52c096e5c7f945ba9c54cb06e0d7833e [id=15c00efe5d7e4f2586f9f0dab975346f]). Expected version 2 got 0
I20260504 13:56:09.288790 9934 ts_tablet_manager.cc:1434] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.196s user 0.027s sys 0.008s
W20260504 13:56:09.293623 9593 tablet.cc:2404] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260504 13:56:09.299377 9934 tablet_bootstrap.cc:492] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:09.306455 9883 tablet.cc:2404] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260504 13:56:09.319767 9934 tablet_bootstrap.cc:654] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:09.322433 9931 tablet_bootstrap.cc:492] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:09.323073 9931 ts_tablet_manager.cc:1403] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.078s user 0.005s sys 0.009s
I20260504 13:56:09.323660 9342 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_cc1e243c7c1941fd9643eff155e02f4f" } modify_external_catalogs: true
I20260504 13:56:09.324476 9342 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_cc1e243c7c1941fd9643eff155e02f4f" } modify_external_catalogs: true
I20260504 13:56:09.328019 9931 raft_consensus.cc:359] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:09.329438 9339 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_b4f9cf9c2ee740929d78f0d29f09ba92" } new_table_name: "default.Table_bb128d99831044cbb85081f8413d182e" modify_external_catalogs: true
I20260504 13:56:09.333144 9931 raft_consensus.cc:385] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:09.333550 9931 raft_consensus.cc:740] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:09.334350 9931 consensus_queue.cc:260] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:09.337093 9338 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_9fb74f3a10f641028449869ce8600210" } new_table_name: "default.Table_02ca7067550b4045804cc5764e09c086" modify_external_catalogs: true
I20260504 13:56:09.346002 9931 ts_tablet_manager.cc:1434] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
W20260504 13:56:09.346261 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet b6182f0cd5474ca69d1ccb4963dd1144 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6182f0cd5474ca69d1ccb4963dd1144 already in progress: creating tablet
I20260504 13:56:09.352059 9931 tablet_bootstrap.cc:492] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:09.362846 9944 tablet_replica.cc:333] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:09.364356 9944 raft_consensus.cc:2243] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:09.373543 9934 tablet_bootstrap.cc:492] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:09.374145 9934 ts_tablet_manager.cc:1403] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.004s
I20260504 13:56:09.383031 9934 raft_consensus.cc:359] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:09.385043 9934 raft_consensus.cc:385] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:09.386070 9934 raft_consensus.cc:740] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:09.387538 9934 consensus_queue.cc:260] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:09.389706 9339 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2951cbd2df4f402eabeef323d121ce1b consensus state has no leader
I20260504 13:56:09.391032 9931 tablet_bootstrap.cc:654] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:09.399108 9934 ts_tablet_manager.cc:1434] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20260504 13:56:09.403294 9944 raft_consensus.cc:2272] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:09.404460 9934 tablet_bootstrap.cc:492] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:09.410104 9338 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 has no consensus state
I20260504 13:56:09.414784 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_d4a59933eb124b4295cc0b066a46f9ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:09.416115 9931 tablet_bootstrap.cc:492] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:09.416527 9931 ts_tablet_manager.cc:1403] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.000s
W20260504 13:56:09.416715 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4a59933eb124b4295cc0b066a46f9ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:09.417025 9880 heartbeater.cc:646] Failed to heartbeat to 127.8.238.253:36337 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111)
I20260504 13:56:09.419070 9814 tablet_service.cc:1558] Processing DeleteTablet for tablet b6182f0cd5474ca69d1ccb4963dd1144 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:09 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:09.419128 9931 raft_consensus.cc:359] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:09.419838 9931 raft_consensus.cc:385] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:09.420140 9931 raft_consensus.cc:740] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:09.421005 9934 tablet_bootstrap.cc:654] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:09.421181 9813 tablet_service.cc:1511] Processing CreateTablet for tablet 5625163f6add4a20b1828905d90b5486 (DEFAULT_TABLE table=default.Table_9242dce1382d42b2b059a7d4b480b5dd [id=7b3ef250728944d9998ef2df032f8819]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.422282 9938 raft_consensus.cc:493] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:09.422724 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5625163f6add4a20b1828905d90b5486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.422847 9938 raft_consensus.cc:515] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:09.426255 9339 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_358b445896d0436d9eebae53ffa9691c" } modify_external_catalogs: true
I20260504 13:56:09.427016 9339 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_358b445896d0436d9eebae53ffa9691c" } modify_external_catalogs: true
I20260504 13:56:09.431490 9931 consensus_queue.cc:260] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:09.436626 9947 raft_consensus.cc:493] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:09.437261 9947 raft_consensus.cc:515] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:09.442193 9947 leader_election.cc:290] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:09.434445 9938 leader_election.cc:290] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:09.458537 9954 tablet_replica.cc:333] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:09.459842 9954 raft_consensus.cc:2243] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:09.461382 9954 raft_consensus.cc:2272] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:09.466918 9954 ts_tablet_manager.cc:1916] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:09.471740 9934 tablet_bootstrap.cc:492] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:09.475893 9934 ts_tablet_manager.cc:1403] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.072s user 0.015s sys 0.007s
I20260504 13:56:09.487010 9934 raft_consensus.cc:359] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:09.493134 9931 ts_tablet_manager.cc:1434] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.076s user 0.010s sys 0.009s
I20260504 13:56:09.497291 9339 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 191a8e5ca677411a9904ff84c1912a71
I20260504 13:56:09.499074 9934 raft_consensus.cc:385] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:09.501336 9934 raft_consensus.cc:740] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:09.502544 9954 ts_tablet_manager.cc:1929] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:09.502074 9934 consensus_queue.cc:260] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:09.507360 9934 ts_tablet_manager.cc:1434] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.030s user 0.008s sys 0.000s
I20260504 13:56:09.507702 9954 log.cc:1199] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/b6182f0cd5474ca69d1ccb4963dd1144
I20260504 13:56:09.508401 9934 tablet_bootstrap.cc:492] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:09.509063 9954 ts_tablet_manager.cc:1950] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:09.517004 9934 tablet_bootstrap.cc:654] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:09.528066 9474 leader_election.cc:343] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Not found: Tablet not found: b6182f0cd5474ca69d1ccb4963dd1144
I20260504 13:56:09.538843 9959 raft_consensus.cc:493] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:09.539469 9959 raft_consensus.cc:515] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:09.542346 9959 leader_election.cc:290] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:09.551635 9934 tablet_bootstrap.cc:492] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:09.552264 9934 ts_tablet_manager.cc:1403] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.044s user 0.014s sys 0.005s
I20260504 13:56:09.557039 9934 raft_consensus.cc:359] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:09.561427 9932 tablet_bootstrap.cc:492] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:09.561986 9932 ts_tablet_manager.cc:1403] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.326s user 0.063s sys 0.119s
I20260504 13:56:09.565379 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet b6182f0cd5474ca69d1ccb4963dd1144 (table default.Table_83c39f83c18b4f6fb7bbbae7136b1c30 [id=fc0aa7190c444823b365c3399bba1aed]) successfully deleted
I20260504 13:56:09.571403 9932 raft_consensus.cc:359] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:09.572098 9932 raft_consensus.cc:385] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:09.572364 9932 raft_consensus.cc:740] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:09.573365 9944 ts_tablet_manager.cc:1916] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:09.577623 9934 raft_consensus.cc:385] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:09.578068 9934 raft_consensus.cc:740] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:09.579384 9931 tablet_bootstrap.cc:492] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:09.578912 9934 consensus_queue.cc:260] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:09.585976 9931 tablet_bootstrap.cc:654] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:09.587911 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_eb48d03e69cd40e69ca8ae3e9f3e9dcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:09.590970 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_211cf2d4e831418f955d214b8f2a8dd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:09.594465 9341 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_bb128d99831044cbb85081f8413d182e" } new_table_name: "default.Table_d6ee06783219496b94ecc1a88d13b0da" modify_external_catalogs: true
I20260504 13:56:09.595669 9931 tablet_bootstrap.cc:492] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:09.596122 9931 ts_tablet_manager.cc:1403] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
W20260504 13:56:09.598346 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb48d03e69cd40e69ca8ae3e9f3e9dcf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:09.598843 9931 raft_consensus.cc:359] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:09.599536 9931 raft_consensus.cc:385] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:09.599805 9931 raft_consensus.cc:740] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
W20260504 13:56:09.601030 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_211cf2d4e831418f955d214b8f2a8dd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:09.600905 9931 consensus_queue.cc:260] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:09.601907 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet c6a6b276cfa14a85a5324728a12b4609 (table default.Table_52c096e5c7f945ba9c54cb06e0d7833e [id=15c00efe5d7e4f2586f9f0dab975346f]). Expected version 2 got 0
I20260504 13:56:09.609311 9666 tablet_service.cc:1511] Processing CreateTablet for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 (DEFAULT_TABLE table=default.Table_02ca7067550b4045804cc5764e09c086 [id=3822050e2a9d4a9db8b0e36430fd0b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.610874 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.611380 9519 tablet_service.cc:1511] Processing CreateTablet for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 (DEFAULT_TABLE table=default.Table_02ca7067550b4045804cc5764e09c086 [id=3822050e2a9d4a9db8b0e36430fd0b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.612532 9519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.616873 9344 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_52c096e5c7f945ba9c54cb06e0d7833e" } new_table_name: "default.Table_7ac44928fca54f5eaadf591188fc5a03" modify_external_catalogs: true
I20260504 13:56:09.619858 9931 ts_tablet_manager.cc:1434] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260504 13:56:09.631258 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_6eb87a052351414aa9f21c6ab20762ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:09.633191 9813 tablet_service.cc:1511] Processing CreateTablet for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 (DEFAULT_TABLE table=default.Table_02ca7067550b4045804cc5764e09c086 [id=3822050e2a9d4a9db8b0e36430fd0b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.634233 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.634802 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6182f0cd5474ca69d1ccb4963dd1144" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:09.635318 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2951cbd2df4f402eabeef323d121ce1b" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:09.635632 9540 raft_consensus.cc:2468] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:09.635913 9539 raft_consensus.cc:2468] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:09.637032 9767 leader_election.cc:304] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
W20260504 13:56:09.637954 9767 raft_consensus.cc:2686] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260504 13:56:09.642459 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6eb87a052351414aa9f21c6ab20762ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:09.643944 9767 leader_election.cc:304] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:09.645167 9934 ts_tablet_manager.cc:1434] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.091s user 0.006s sys 0.000s
I20260504 13:56:09.645756 9959 raft_consensus.cc:2804] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:09.646109 9959 raft_consensus.cc:493] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:09.646456 9959 raft_consensus.cc:3060] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:09.647886 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6182f0cd5474ca69d1ccb4963dd1144" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:09.652207 9934 tablet_bootstrap.cc:492] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:09.652683 9473 leader_election.cc:343] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: Tablet not RUNNING: STOPPED
I20260504 13:56:09.653080 9473 leader_election.cc:304] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:09.653338 9959 raft_consensus.cc:515] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:09.653967 9947 raft_consensus.cc:2749] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:09.658329 9934 tablet_bootstrap.cc:654] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:09.658824 9336 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_b10106d800a645c5969c4434f58a43ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:09.573025 9932 consensus_queue.cc:260] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:09.660688 9336 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b10106d800a645c5969c4434f58a43ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:09.674057 9932 ts_tablet_manager.cc:1434] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.112s user 0.007s sys 0.000s
I20260504 13:56:09.687683 9932 tablet_bootstrap.cc:492] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:09.687896 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet c6a6b276cfa14a85a5324728a12b4609 (table default.Table_52c096e5c7f945ba9c54cb06e0d7833e [id=15c00efe5d7e4f2586f9f0dab975346f]). Expected version 2 got 0
I20260504 13:56:09.700027 9934 tablet_bootstrap.cc:492] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:09.700517 9934 ts_tablet_manager.cc:1403] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.049s user 0.013s sys 0.000s
I20260504 13:56:09.700598 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 2951cbd2df4f402eabeef323d121ce1b (table default.Table_bb128d99831044cbb85081f8413d182e [id=46ac1f0baae0492984e87927e4905343]). Expected version 2 got 0
I20260504 13:56:09.703594 9934 raft_consensus.cc:359] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:09.704355 9934 raft_consensus.cc:385] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:09.704690 9934 raft_consensus.cc:740] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
W20260504 13:56:09.714149 9341 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2951cbd2df4f402eabeef323d121ce1b consensus state has no leader
I20260504 13:56:09.714237 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2951cbd2df4f402eabeef323d121ce1b" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:09.715967 9539 raft_consensus.cc:3060] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:09.716665 9342 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 357407221df147df95940c492a38f3e8
I20260504 13:56:09.714921 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2951cbd2df4f402eabeef323d121ce1b" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:09.718963 9664 tablet_service.cc:1558] Processing DeleteTablet for tablet 357407221df147df95940c492a38f3e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:09 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:09.719038 9814 tablet_service.cc:1558] Processing DeleteTablet for tablet 357407221df147df95940c492a38f3e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:09 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:09.719753 9954 tablet_replica.cc:333] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:09.720505 9954 raft_consensus.cc:2243] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:09.720991 9954 raft_consensus.cc:2272] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:09.720867 9768 leader_election.cc:343] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:09.722429 9519 tablet_service.cc:1558] Processing DeleteTablet for tablet 357407221df147df95940c492a38f3e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:09 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:09.723270 9945 tablet_replica.cc:333] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:09.724050 9945 raft_consensus.cc:2243] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:09.724624 9945 raft_consensus.cc:2272] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:09.727587 9954 ts_tablet_manager.cc:1916] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:09.729650 9945 ts_tablet_manager.cc:1916] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:09.732291 9341 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_ba0d3ea2e915469cbb3737d5d7d1a56d" } modify_external_catalogs: true
I20260504 13:56:09.735126 9341 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_ba0d3ea2e915469cbb3737d5d7d1a56d" } modify_external_catalogs: true
I20260504 13:56:09.716156 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6182f0cd5474ca69d1ccb4963dd1144" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:09.742638 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2951cbd2df4f402eabeef323d121ce1b" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:09.743271 9947 raft_consensus.cc:493] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 13:56:09.743206 9768 leader_election.cc:343] T b6182f0cd5474ca69d1ccb4963dd1144 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: Tablet not RUNNING: STOPPED
I20260504 13:56:09.743729 9947 raft_consensus.cc:515] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:09.745507 9947 leader_election.cc:290] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:09.747030 9539 raft_consensus.cc:2468] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:09.747468 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6a6b276cfa14a85a5324728a12b4609" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:09.748181 9686 raft_consensus.cc:2468] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:09.748277 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6a6b276cfa14a85a5324728a12b4609" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:09.749262 9473 leader_election.cc:304] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:09.749294 9834 raft_consensus.cc:2468] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:09.749979 9947 raft_consensus.cc:2804] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:09.750291 9947 raft_consensus.cc:493] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:09.750599 9947 raft_consensus.cc:3060] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:09.750633 9767 leader_election.cc:304] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:09.728859 9959 leader_election.cc:290] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:09.751725 9959 raft_consensus.cc:2804] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
W20260504 13:56:09.754611 9768 leader_election.cc:343] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:09.756017 9959 raft_consensus.cc:697] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
W20260504 13:56:09.756134 9344 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6a6b276cfa14a85a5324728a12b4609 consensus state has no leader
I20260504 13:56:09.755997 9337 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 2951cbd2df4f402eabeef323d121ce1b (table default.Table_d6ee06783219496b94ecc1a88d13b0da [id=46ac1f0baae0492984e87927e4905343]). Expected version 3 got 0
I20260504 13:56:09.757014 9959 consensus_queue.cc:237] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:09.705407 9934 consensus_queue.cc:260] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:09.777218 9954 ts_tablet_manager.cc:1929] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:09.790661 9519 tablet_service.cc:1558] Processing DeleteTablet for tablet b6182f0cd5474ca69d1ccb4963dd1144 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:09 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:09.800535 9934 ts_tablet_manager.cc:1434] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.100s user 0.006s sys 0.005s
I20260504 13:56:09.800698 9932 tablet_bootstrap.cc:654] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:09.804010 9934 tablet_bootstrap.cc:492] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:09.808318 9954 log.cc:1199] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/357407221df147df95940c492a38f3e8
I20260504 13:56:09.809126 9954 ts_tablet_manager.cc:1950] T 357407221df147df95940c492a38f3e8 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:09.811750 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 357407221df147df95940c492a38f3e8 (table default.Table_cc1e243c7c1941fd9643eff155e02f4f [id=4ae0e62b5bad4723a28421c4aba76e5d]) successfully deleted
I20260504 13:56:09.814059 9945 ts_tablet_manager.cc:1929] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:09.815136 9947 raft_consensus.cc:515] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:09.816720 9934 tablet_bootstrap.cc:654] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:09.823452 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6a6b276cfa14a85a5324728a12b4609" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:09.823949 9834 raft_consensus.cc:3060] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:09.830550 9834 raft_consensus.cc:2468] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:09.831506 9931 tablet_bootstrap.cc:492] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:09.831665 9474 leader_election.cc:304] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:09.834299 9947 leader_election.cc:290] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:09.834797 9947 raft_consensus.cc:2804] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:09.835409 9947 raft_consensus.cc:697] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:09.836822 9947 consensus_queue.cc:237] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:09.838340 9931 tablet_bootstrap.cc:654] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:09.833917 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6a6b276cfa14a85a5324728a12b4609" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:09.840296 9686 raft_consensus.cc:3060] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:09.842335 9934 tablet_bootstrap.cc:492] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:09.842801 9934 ts_tablet_manager.cc:1403] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.001s
I20260504 13:56:09.847878 9686 raft_consensus.cc:2468] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
W20260504 13:56:09.851332 9328 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet c6a6b276cfa14a85a5324728a12b4609 (table default.Table_7ac44928fca54f5eaadf591188fc5a03 [id=15c00efe5d7e4f2586f9f0dab975346f], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet c6a6b276cfa14a85a5324728a12b4609 consensus state has no leader
I20260504 13:56:09.850708 9934 raft_consensus.cc:359] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:09.853286 9934 raft_consensus.cc:385] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:09.853621 9934 raft_consensus.cc:740] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:09.854362 9934 consensus_queue.cc:260] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:09.857367 9944 ts_tablet_manager.cc:1929] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:09.862812 9934 ts_tablet_manager.cc:1434] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.020s user 0.001s sys 0.004s
I20260504 13:56:09.859583 9945 log.cc:1199] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/357407221df147df95940c492a38f3e8
I20260504 13:56:09.864099 9945 ts_tablet_manager.cc:1950] T 357407221df147df95940c492a38f3e8 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:09.867121 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 357407221df147df95940c492a38f3e8 (table default.Table_cc1e243c7c1941fd9643eff155e02f4f [id=4ae0e62b5bad4723a28421c4aba76e5d]) successfully deleted
I20260504 13:56:09.870647 9931 tablet_bootstrap.cc:492] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:09.870648 9341 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 432a7f95486b4a3ba980745a27a86138
I20260504 13:56:09.871122 9945 tablet_replica.cc:333] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:09.871105 9931 ts_tablet_manager.cc:1403] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.000s
I20260504 13:56:09.873991 9931 raft_consensus.cc:359] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:09.874747 9931 raft_consensus.cc:385] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:09.875077 9931 raft_consensus.cc:740] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:09.875880 9931 consensus_queue.cc:260] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:09.871989 9945 raft_consensus.cc:2243] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:09.877144 9945 raft_consensus.cc:2272] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:09.882144 9944 log.cc:1199] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/b6182f0cd5474ca69d1ccb4963dd1144
I20260504 13:56:09.883211 9944 ts_tablet_manager.cc:1950] T b6182f0cd5474ca69d1ccb4963dd1144 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:09.891155 9945 ts_tablet_manager.cc:1916] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:09.896288 9932 tablet_bootstrap.cc:492] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:09.896802 9932 ts_tablet_manager.cc:1403] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.209s user 0.007s sys 0.005s
I20260504 13:56:09.899714 9932 raft_consensus.cc:359] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:09.905081 9340 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 2951cbd2df4f402eabeef323d121ce1b (table default.Table_d6ee06783219496b94ecc1a88d13b0da [id=46ac1f0baae0492984e87927e4905343]). Expected version 3 got 0
I20260504 13:56:09.907681 9813 tablet_service.cc:1511] Processing CreateTablet for tablet 52ff55bef5e54353ab9ca09130ab1e0e (DEFAULT_TABLE table=default.Table_d4a59933eb124b4295cc0b066a46f9ea [id=493e2db02edb42c79460ec701aa92679]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.908731 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ff55bef5e54353ab9ca09130ab1e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.908404 9343 catalog_manager.cc:5671] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:09.910499 9343 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 2951cbd2df4f402eabeef323d121ce1b (table default.Table_d6ee06783219496b94ecc1a88d13b0da [id=46ac1f0baae0492984e87927e4905343]). Expected version 3 got 0
I20260504 13:56:09.916115 9945 ts_tablet_manager.cc:1929] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:09.916561 9945 log.cc:1199] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/b6182f0cd5474ca69d1ccb4963dd1144
I20260504 13:56:09.917276 9945 ts_tablet_manager.cc:1950] T b6182f0cd5474ca69d1ccb4963dd1144 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:09.919661 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet b6182f0cd5474ca69d1ccb4963dd1144 (table default.Table_83c39f83c18b4f6fb7bbbae7136b1c30 [id=fc0aa7190c444823b365c3399bba1aed]) successfully deleted
I20260504 13:56:09.921788 9340 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_b10106d800a645c5969c4434f58a43ac" } new_table_name: "default.Table_c943f49871964cdc9c802e13bfb75928" modify_external_catalogs: true
I20260504 13:56:09.924767 9931 ts_tablet_manager.cc:1434] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.053s user 0.003s sys 0.002s
I20260504 13:56:09.927685 9519 tablet_service.cc:1511] Processing CreateTablet for tablet 52ff55bef5e54353ab9ca09130ab1e0e (DEFAULT_TABLE table=default.Table_d4a59933eb124b4295cc0b066a46f9ea [id=493e2db02edb42c79460ec701aa92679]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.928736 9519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ff55bef5e54353ab9ca09130ab1e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.932163 9341 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_6eb87a052351414aa9f21c6ab20762ce" } modify_external_catalogs: true
I20260504 13:56:09.932687 9341 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_6eb87a052351414aa9f21c6ab20762ce" } modify_external_catalogs: true
I20260504 13:56:09.935034 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet b6182f0cd5474ca69d1ccb4963dd1144 (table default.Table_83c39f83c18b4f6fb7bbbae7136b1c30 [id=fc0aa7190c444823b365c3399bba1aed]) successfully deleted
I20260504 13:56:09.939047 9336 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_c9c66820c2c24a1795b44da8856630d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:09.940732 9336 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9c66820c2c24a1795b44da8856630d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:09.945902 9339 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 (table default.Table_02ca7067550b4045804cc5764e09c086 [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 1 got 0
I20260504 13:56:09.915132 9666 tablet_service.cc:1511] Processing CreateTablet for tablet 52ff55bef5e54353ab9ca09130ab1e0e (DEFAULT_TABLE table=default.Table_d4a59933eb124b4295cc0b066a46f9ea [id=493e2db02edb42c79460ec701aa92679]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.928727 9664 tablet_service.cc:1511] Processing CreateTablet for tablet 385cc1f6f4b24c268c25c61a4741480d (DEFAULT_TABLE table=default.Table_5d8e09780e014609b010c3b0b49726eb [id=0b1606cb64b440cea0b713bd369c440d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.953227 9664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385cc1f6f4b24c268c25c61a4741480d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.935873 9944 tablet_replica.cc:333] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:09.959658 9934 tablet_bootstrap.cc:492] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:09.965737 9934 tablet_bootstrap.cc:654] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:09.975291 9341 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 82d65dc083ab4892ae9486cf8b011d26
I20260504 13:56:09.977664 9931 tablet_bootstrap.cc:492] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:09.981453 9339 catalog_manager.cc:5671] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:09.982739 9339 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet c6a6b276cfa14a85a5324728a12b4609 (table default.Table_7ac44928fca54f5eaadf591188fc5a03 [id=15c00efe5d7e4f2586f9f0dab975346f]). Expected version 3 got 0
I20260504 13:56:09.986289 9814 tablet_service.cc:1511] Processing CreateTablet for tablet 385cc1f6f4b24c268c25c61a4741480d (DEFAULT_TABLE table=default.Table_5d8e09780e014609b010c3b0b49726eb [id=0b1606cb64b440cea0b713bd369c440d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:09.987370 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385cc1f6f4b24c268c25c61a4741480d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:09.983139 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_895e961d6c1a45e688ad4e0ca752bae9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:09.988639 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_895e961d6c1a45e688ad4e0ca752bae9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:09.987874 9339 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 (table default.Table_02ca7067550b4045804cc5764e09c086 [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 1 got 0
I20260504 13:56:09.983770 9343 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_211cf2d4e831418f955d214b8f2a8dd2" } new_table_name: "default.Table_00c39fd2584d48fb965f7e81908e2ae7" modify_external_catalogs: true
W20260504 13:56:09.987334 9340 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d25c16864f343a88fa77535831d5a52 has no consensus state
I20260504 13:56:09.996297 9934 tablet_bootstrap.cc:492] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:09.996692 9934 ts_tablet_manager.cc:1403] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.004s
I20260504 13:56:09.998958 9519 tablet_service.cc:1511] Processing CreateTablet for tablet 385cc1f6f4b24c268c25c61a4741480d (DEFAULT_TABLE table=default.Table_5d8e09780e014609b010c3b0b49726eb [id=0b1606cb64b440cea0b713bd369c440d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:10.000046 9519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385cc1f6f4b24c268c25c61a4741480d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:10.003785 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ff55bef5e54353ab9ca09130ab1e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:10.007674 9931 tablet_bootstrap.cc:654] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:10.007493 9934 raft_consensus.cc:359] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.015267 9934 raft_consensus.cc:385] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:10.015618 9934 raft_consensus.cc:740] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:10.017936 9934 consensus_queue.cc:260] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.020134 9931 tablet_bootstrap.cc:492] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:10.020519 9931 ts_tablet_manager.cc:1403] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.003s
I20260504 13:56:10.023281 9932 raft_consensus.cc:385] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:10.023716 9932 raft_consensus.cc:740] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
W20260504 13:56:10.024911 9768 consensus_peers.cc:597] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:10.025614 9932 consensus_queue.cc:260] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:10.029078 9539 raft_consensus.cc:1275] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Refusing update from remote peer d4051d2bf0bf410eb8acd107560df888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 13:56:10.030571 9934 ts_tablet_manager.cc:1434] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20260504 13:56:10.030646 9959 consensus_queue.cc:1048] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:10.038023 9934 tablet_bootstrap.cc:492] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:10.043159 9944 raft_consensus.cc:2243] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:10.043488 9934 tablet_bootstrap.cc:654] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:10.043151 9931 raft_consensus.cc:359] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.043735 9944 raft_consensus.cc:2272] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:10.045387 9931 raft_consensus.cc:385] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:10.046007 9931 raft_consensus.cc:740] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:10.047040 9931 consensus_queue.cc:260] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.059831 9944 ts_tablet_manager.cc:1916] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:10.061329 9931 ts_tablet_manager.cc:1434] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20260504 13:56:10.068704 9931 tablet_bootstrap.cc:492] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:10.068092 9932 ts_tablet_manager.cc:1434] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.171s user 0.004s sys 0.002s
I20260504 13:56:10.074635 9335 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_7ac44928fca54f5eaadf591188fc5a03" } modify_external_catalogs: true
I20260504 13:56:10.075193 9335 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_7ac44928fca54f5eaadf591188fc5a03" } modify_external_catalogs: true
I20260504 13:56:10.075868 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_c9e39aac400445028318ffebeb95209d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:10.075155 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_4227488cd29f4bcf84adca4a41b1f73f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:10.077611 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4227488cd29f4bcf84adca4a41b1f73f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:10.077632 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9e39aac400445028318ffebeb95209d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:10.101806 9340 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 2951cbd2df4f402eabeef323d121ce1b (table default.Table_d6ee06783219496b94ecc1a88d13b0da [id=46ac1f0baae0492984e87927e4905343]). Expected version 3 got 0
I20260504 13:56:10.133499 9931 tablet_bootstrap.cc:654] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:10.138031 9977 mvcc.cc:204] Tried to move back new op lower bound from 7282290565186719744 to 7282290564146556928. Current Snapshot: MvccSnapshot[applied={T|T < 7282290565186719744}]
I20260504 13:56:10.150866 9931 tablet_bootstrap.cc:492] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:10.151511 9931 ts_tablet_manager.cc:1403] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.083s user 0.022s sys 0.000s
I20260504 13:56:10.152513 9932 tablet_bootstrap.cc:492] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:10.158989 9931 raft_consensus.cc:359] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.161970 9931 raft_consensus.cc:385] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:10.163127 9931 raft_consensus.cc:740] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:10.166136 9931 consensus_queue.cc:260] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.173321 9340 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 2951cbd2df4f402eabeef323d121ce1b (table default.Table_d6ee06783219496b94ecc1a88d13b0da [id=46ac1f0baae0492984e87927e4905343]). Expected version 3 got 0
I20260504 13:56:10.187624 9931 ts_tablet_manager.cc:1434] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.036s user 0.010s sys 0.000s
I20260504 13:56:10.196565 9336 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_5d8e09780e014609b010c3b0b49726eb" } new_table_name: "default.Table_659330a3d91d404ebc0d526364ed2f0b" modify_external_catalogs: true
I20260504 13:56:10.207068 9340 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_c943f49871964cdc9c802e13bfb75928" } modify_external_catalogs: true
I20260504 13:56:10.209719 9340 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_c943f49871964cdc9c802e13bfb75928" } modify_external_catalogs: true
I20260504 13:56:10.212414 9982 tablet.cc:1724] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260504 13:56:10.212625 9980 tablet.cc:1724] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260504 13:56:10.218963 9938 raft_consensus.cc:493] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:10.219645 9938 raft_consensus.cc:515] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:10.224498 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf2b59d13da40a2b2160ebdc0b2eb80" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:10.225852 9938 leader_election.cc:290] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:10.227079 9768 leader_election.cc:343] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:10.229846 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf2b59d13da40a2b2160ebdc0b2eb80" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:10.230696 9540 raft_consensus.cc:2468] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:10.232386 9767 leader_election.cc:304] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:10.233952 9938 raft_consensus.cc:2804] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:10.234500 9938 raft_consensus.cc:493] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:10.235100 9938 raft_consensus.cc:3060] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:10.253047 9343 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 39f080a0296b44988e8e51e68453a551 has no consensus state
I20260504 13:56:10.263908 9932 tablet_bootstrap.cc:654] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:10.264722 9975 mvcc.cc:204] Tried to move back new op lower bound from 7282290565186719744 to 7282290564146556928. Current Snapshot: MvccSnapshot[applied={T|T < 7282290565186719744}]
I20260504 13:56:10.264215 9934 tablet_bootstrap.cc:492] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:10.266538 9934 ts_tablet_manager.cc:1403] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.229s user 0.053s sys 0.088s
I20260504 13:56:10.268815 9934 raft_consensus.cc:359] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.269407 9934 raft_consensus.cc:385] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:10.269652 9934 raft_consensus.cc:740] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:10.270184 9934 consensus_queue.cc:260] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.274585 9934 ts_tablet_manager.cc:1434] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260504 13:56:10.303706 9980 tablet.cc:1719] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888: Skipping requested alter to schema version 3, tablet already version 3
I20260504 13:56:10.304456 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_7b3c23577d974ef0bdf80b361e3738c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:10.306052 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b3c23577d974ef0bdf80b361e3738c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:10.308415 9947 raft_consensus.cc:493] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:10.309051 9947 raft_consensus.cc:515] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:10.312732 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5625163f6add4a20b1828905d90b5486" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
W20260504 13:56:10.315989 9473 leader_election.cc:343] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:10.317265 9944 ts_tablet_manager.cc:1929] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:10.317644 9944 log.cc:1199] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/357407221df147df95940c492a38f3e8
I20260504 13:56:10.318264 9944 ts_tablet_manager.cc:1950] T 357407221df147df95940c492a38f3e8 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:10.319263 9947 leader_election.cc:290] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:10.320397 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5625163f6add4a20b1828905d90b5486" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:10.320948 9932 tablet_bootstrap.cc:492] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:10.320976 9834 raft_consensus.cc:2468] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:10.321357 9932 ts_tablet_manager.cc:1403] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.169s user 0.010s sys 0.000s
I20260504 13:56:10.322131 9474 leader_election.cc:304] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:10.323747 9932 raft_consensus.cc:359] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:10.324412 9932 raft_consensus.cc:385] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:10.324682 9932 raft_consensus.cc:740] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:10.325299 9932 consensus_queue.cc:260] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:10.325778 9968 raft_consensus.cc:2804] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:10.326089 9968 raft_consensus.cc:493] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:10.326390 9968 raft_consensus.cc:3060] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:10.327430 9932 ts_tablet_manager.cc:1434] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.006s user 0.005s sys 0.003s
I20260504 13:56:10.328294 9932 tablet_bootstrap.cc:492] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:10.344677 9336 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 385cc1f6f4b24c268c25c61a4741480d consensus state has no leader
I20260504 13:56:10.346230 9932 tablet_bootstrap.cc:654] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:10.349995 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 357407221df147df95940c492a38f3e8 (table default.Table_cc1e243c7c1941fd9643eff155e02f4f [id=4ae0e62b5bad4723a28421c4aba76e5d]) successfully deleted
I20260504 13:56:10.355278 9686 raft_consensus.cc:1275] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Refusing update from remote peer 7b79fe7dd8934dd996393a2d96eabf00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 13:56:10.358850 9834 raft_consensus.cc:1275] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Refusing update from remote peer 7b79fe7dd8934dd996393a2d96eabf00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 13:56:10.363395 9947 consensus_queue.cc:1048] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:10.366086 9932 tablet_bootstrap.cc:492] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:10.366931 9932 ts_tablet_manager.cc:1403] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.006s
I20260504 13:56:10.369524 9932 raft_consensus.cc:359] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:10.370201 9932 raft_consensus.cc:385] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:10.370491 9932 raft_consensus.cc:740] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:10.371243 9932 consensus_queue.cc:260] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:10.377671 9990 consensus_queue.cc:1048] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:10.380795 9932 ts_tablet_manager.cc:1434] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260504 13:56:10.381675 9932 tablet_bootstrap.cc:492] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:10.387627 9932 tablet_bootstrap.cc:654] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:10.389330 9968 raft_consensus.cc:515] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:10.420159 9343 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_d5cc86d81fb043319576af0a39c643a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:10.422590 9343 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5cc86d81fb043319576af0a39c643a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:10.430699 9344 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 385cc1f6f4b24c268c25c61a4741480d (table default.Table_659330a3d91d404ebc0d526364ed2f0b [id=0b1606cb64b440cea0b713bd369c440d]). Expected version 1 got 0
I20260504 13:56:10.450446 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5625163f6add4a20b1828905d90b5486" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:10.451193 9685 raft_consensus.cc:3060] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:10.459415 9338 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_9242dce1382d42b2b059a7d4b480b5dd" } new_table_name: "default.Table_56891f8543a84e859c39b381625a9daa" modify_external_catalogs: true
I20260504 13:56:10.434211 9947 raft_consensus.cc:493] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:10.462687 9947 raft_consensus.cc:515] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.467041 9938 raft_consensus.cc:515] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:10.475618 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ff55bef5e54353ab9ca09130ab1e0e" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
W20260504 13:56:10.477799 9473 leader_election.cc:343] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:10.480167 9685 raft_consensus.cc:2468] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:10.482002 9968 leader_election.cc:290] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:10.484992 9336 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_7e28164fb19d47649ad4454da7cc5f85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:10.487993 9336 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e28164fb19d47649ad4454da7cc5f85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:10.498653 9664 tablet_service.cc:1511] Processing CreateTablet for tablet 8d25c16864f343a88fa77535831d5a52 (DEFAULT_TABLE table=default.Table_c943f49871964cdc9c802e13bfb75928 [id=ed4c94257e234052adc4d311758ebbd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:10.547778 9664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d25c16864f343a88fa77535831d5a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:10.539867 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf2b59d13da40a2b2160ebdc0b2eb80" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:10.554035 9684 raft_consensus.cc:3060] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:10.568532 9813 tablet_service.cc:1511] Processing CreateTablet for tablet 8d25c16864f343a88fa77535831d5a52 (DEFAULT_TABLE table=default.Table_c943f49871964cdc9c802e13bfb75928 [id=ed4c94257e234052adc4d311758ebbd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:10.569597 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d25c16864f343a88fa77535831d5a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:10.573215 9684 raft_consensus.cc:2468] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:10.574425 9768 leader_election.cc:304] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:10.575814 9983 raft_consensus.cc:2804] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:10.589044 9666 tablet_service.cc:1511] Processing CreateTablet for tablet 39f080a0296b44988e8e51e68453a551 (DEFAULT_TABLE table=default.Table_00c39fd2584d48fb965f7e81908e2ae7 [id=532ece9a0c134c1f8c1c3793638f34c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:10.590637 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39f080a0296b44988e8e51e68453a551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:10.593137 9932 tablet_bootstrap.cc:492] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:10.593530 9932 ts_tablet_manager.cc:1403] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.212s user 0.067s sys 0.082s
I20260504 13:56:10.596097 9932 raft_consensus.cc:359] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.596756 9932 raft_consensus.cc:385] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:10.597046 9932 raft_consensus.cc:740] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:10.597641 9932 consensus_queue.cc:260] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.598258 9814 tablet_service.cc:1511] Processing CreateTablet for tablet 39f080a0296b44988e8e51e68453a551 (DEFAULT_TABLE table=default.Table_00c39fd2584d48fb965f7e81908e2ae7 [id=532ece9a0c134c1f8c1c3793638f34c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:10.599427 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39f080a0296b44988e8e51e68453a551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:10.599650 9932 ts_tablet_manager.cc:1434] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260504 13:56:10.600466 9932 tablet_bootstrap.cc:492] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:10.602733 9931 tablet_bootstrap.cc:492] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:10.605805 9932 tablet_bootstrap.cc:654] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:10.616807 9665 tablet_service.cc:1511] Processing CreateTablet for tablet ddd9d68a1e9d45c9b6f9c8a90912ef54 (DEFAULT_TABLE table=default.Table_eb48d03e69cd40e69ca8ae3e9f3e9dcf [id=0d9014419f714d2c82f3a861926fa0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:10.617822 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddd9d68a1e9d45c9b6f9c8a90912ef54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:10.618027 9931 tablet_bootstrap.cc:654] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:10.629765 9983 raft_consensus.cc:697] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:10.630613 9983 consensus_queue.cc:237] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:10.634397 9813 tablet_service.cc:1511] Processing CreateTablet for tablet ddd9d68a1e9d45c9b6f9c8a90912ef54 (DEFAULT_TABLE table=default.Table_eb48d03e69cd40e69ca8ae3e9f3e9dcf [id=0d9014419f714d2c82f3a861926fa0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:10.635751 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddd9d68a1e9d45c9b6f9c8a90912ef54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:10.664705 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_4bdb8491ef7e4ae487fc734f646abf1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:10.666916 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bdb8491ef7e4ae487fc734f646abf1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:10.676512 9338 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5625163f6add4a20b1828905d90b5486 consensus state has no leader
I20260504 13:56:10.678164 9337 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_4227488cd29f4bcf84adca4a41b1f73f" } new_table_name: "default.Table_15bf9089e68d4882838fe4f1c65156ef" modify_external_catalogs: true
I20260504 13:56:10.531747 9947 leader_election.cc:290] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:10.686616 9520 tablet_service.cc:1511] Processing CreateTablet for tablet 39f080a0296b44988e8e51e68453a551 (DEFAULT_TABLE table=default.Table_00c39fd2584d48fb965f7e81908e2ae7 [id=532ece9a0c134c1f8c1c3793638f34c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:10.688439 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39f080a0296b44988e8e51e68453a551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:10.688212 9518 tablet_service.cc:1511] Processing CreateTablet for tablet ddd9d68a1e9d45c9b6f9c8a90912ef54 (DEFAULT_TABLE table=default.Table_eb48d03e69cd40e69ca8ae3e9f3e9dcf [id=0d9014419f714d2c82f3a861926fa0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:10.686625 9519 tablet_service.cc:1511] Processing CreateTablet for tablet 8d25c16864f343a88fa77535831d5a52 (DEFAULT_TABLE table=default.Table_c943f49871964cdc9c802e13bfb75928 [id=ed4c94257e234052adc4d311758ebbd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:10.692283 9519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d25c16864f343a88fa77535831d5a52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:10.702394 9473 leader_election.cc:304] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:10.703352 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddd9d68a1e9d45c9b6f9c8a90912ef54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:10.703435 9947 raft_consensus.cc:2804] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:10.708418 9947 raft_consensus.cc:697] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:10.709404 9947 consensus_queue.cc:237] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:10.735095 9931 tablet_bootstrap.cc:492] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:10.735531 9931 ts_tablet_manager.cc:1403] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.133s user 0.011s sys 0.000s
I20260504 13:56:10.737802 9931 raft_consensus.cc:359] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:10.738452 9931 raft_consensus.cc:385] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:10.738741 9931 raft_consensus.cc:740] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:10.739411 9931 consensus_queue.cc:260] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:10.737407 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_e9289c25896a417c9d71ea71d1242761"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:10.748692 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9289c25896a417c9d71ea71d1242761 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:10.754122 9931 ts_tablet_manager.cc:1434] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260504 13:56:10.754700 9343 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_2099388b6e2a455295be76a6d072758e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:10.756481 9343 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2099388b6e2a455295be76a6d072758e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:10.540653 9938 leader_election.cc:290] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:10.769069 9931 tablet_bootstrap.cc:492] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:10.774120 9931 tablet_bootstrap.cc:654] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:10.780792 9340 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 8d25c16864f343a88fa77535831d5a52
I20260504 13:56:10.782393 9665 tablet_service.cc:1558] Processing DeleteTablet for tablet 8d25c16864f343a88fa77535831d5a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:10 UTC) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:10.788029 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 8d25c16864f343a88fa77535831d5a52 because the tablet was not found. No further retry: Not found: Tablet not found: 8d25c16864f343a88fa77535831d5a52
I20260504 13:56:10.791728 9517 tablet_service.cc:1558] Processing DeleteTablet for tablet 8d25c16864f343a88fa77535831d5a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:10 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:10.792989 9344 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: default.Table_d6ee06783219496b94ecc1a88d13b0da [id=46ac1f0baae0492984e87927e4905343] alter complete (version 3)
I20260504 13:56:10.803887 9814 tablet_service.cc:1558] Processing DeleteTablet for tablet 8d25c16864f343a88fa77535831d5a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:10 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:10.810449 9982 tablet.cc:1724] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260504 13:56:10.813648 10003 tablet_replica.cc:333] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:10.814435 10003 raft_consensus.cc:2243] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:10.814850 10003 raft_consensus.cc:2272] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:10.832463 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 470418us
I20260504 13:56:10.833175 10003 ts_tablet_manager.cc:1916] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:10.834277 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5625163f6add4a20b1828905d90b5486" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:10.834782 9834 raft_consensus.cc:3060] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:10.839787 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ff55bef5e54353ab9ca09130ab1e0e" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:10.840370 9833 raft_consensus.cc:2468] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:10.843994 9684 raft_consensus.cc:3060] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:10.851006 9337 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4dca0cadd5e400eab1365c03a8de4f4 has no consensus state
I20260504 13:56:10.853685 9834 raft_consensus.cc:2468] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:10.828153 9993 mvcc.cc:204] Tried to move back new op lower bound from 7282290566445883392 to 7282290564629430272. Current Snapshot: MvccSnapshot[applied={T|T < 7282290566445883392}]
I20260504 13:56:10.860440 9931 tablet_bootstrap.cc:492] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:10.860877 9931 ts_tablet_manager.cc:1403] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.092s user 0.006s sys 0.002s
W20260504 13:56:10.863788 9331 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 5625163f6add4a20b1828905d90b5486 (table default.Table_56891f8543a84e859c39b381625a9daa [id=7b3ef250728944d9998ef2df032f8819], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5625163f6add4a20b1828905d90b5486 consensus state has no leader
I20260504 13:56:10.864300 9947 raft_consensus.cc:493] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:10.864722 9947 raft_consensus.cc:515] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.869886 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "385cc1f6f4b24c268c25c61a4741480d" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
W20260504 13:56:10.871111 9473 leader_election.cc:343] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:10.863130 9931 raft_consensus.cc:359] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:10.887828 9932 tablet_bootstrap.cc:492] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:10.888270 9932 ts_tablet_manager.cc:1403] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.288s user 0.053s sys 0.056s
I20260504 13:56:10.887784 10004 tablet.cc:1724] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260504 13:56:10.889052 9982 tablet.cc:1719] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00: Skipping requested alter to schema version 3, tablet already version 3
I20260504 13:56:10.894896 9342 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_e9289c25896a417c9d71ea71d1242761" } new_table_name: "default.Table_512c72ff320944448051ca113741e6a4" modify_external_catalogs: true
I20260504 13:56:10.902025 9931 raft_consensus.cc:385] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:10.902290 9931 raft_consensus.cc:740] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:10.902925 9931 consensus_queue.cc:260] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:10.907397 9932 raft_consensus.cc:359] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.907898 9931 ts_tablet_manager.cc:1434] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.047s user 0.007s sys 0.000s
I20260504 13:56:10.908020 9932 raft_consensus.cc:385] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:10.910015 9932 raft_consensus.cc:740] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:10.910657 9932 consensus_queue.cc:260] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.911550 10007 tablet.cc:1724] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260504 13:56:10.912855 9932 ts_tablet_manager.cc:1434] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260504 13:56:10.913688 9932 tablet_bootstrap.cc:492] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:10.922724 9932 tablet_bootstrap.cc:654] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:10.931452 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf2b59d13da40a2b2160ebdc0b2eb80" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:10.932134 9539 raft_consensus.cc:3060] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:10.932552 9941 raft_consensus.cc:493] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:10.933007 9941 raft_consensus.cc:515] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.936412 9947 leader_election.cc:290] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:10.941463 9539 raft_consensus.cc:2468] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
W20260504 13:56:10.942034 9342 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dcc02d5f590e420987daab92fdf8b10a has no consensus state
I20260504 13:56:10.942055 9941 leader_election.cc:290] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:10.937937 9474 leader_election.cc:304] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:10.946900 9947 raft_consensus.cc:2804] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:10.951644 9931 tablet_bootstrap.cc:492] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:10.947693 9947 raft_consensus.cc:493] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:10.954665 9947 raft_consensus.cc:3060] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:10.961798 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "385cc1f6f4b24c268c25c61a4741480d" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:10.962514 9833 raft_consensus.cc:2468] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
W20260504 13:56:10.966145 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 8d25c16864f343a88fa77535831d5a52 because the tablet was not found. No further retry: Not found: Tablet not found: 8d25c16864f343a88fa77535831d5a52
I20260504 13:56:10.969034 9474 leader_election.cc:304] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:10.969909 9968 raft_consensus.cc:2804] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:10.970211 9968 raft_consensus.cc:493] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:10.970510 9968 raft_consensus.cc:3060] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:10.976734 9968 raft_consensus.cc:515] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:10.979357 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "385cc1f6f4b24c268c25c61a4741480d" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:10.979861 9685 raft_consensus.cc:3060] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:10.978358 9338 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 (table default.Table_02ca7067550b4045804cc5764e09c086 [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 1 got 0
I20260504 13:56:10.982393 9335 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet c6a6b276cfa14a85a5324728a12b4609
I20260504 13:56:10.985100 9344 catalog_manager.cc:5671] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:10.986512 9344 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 (table default.Table_02ca7067550b4045804cc5764e09c086 [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 1 got 0
W20260504 13:56:10.988903 9339 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:38691 (request call id 16) took 1020 ms (client timeout 1000 ms). Trace:
I20260504 13:56:10.992300 9813 tablet_service.cc:1558] Processing DeleteTablet for tablet c6a6b276cfa14a85a5324728a12b4609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:10 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:10.994035 9664 tablet_service.cc:1558] Processing DeleteTablet for tablet c6a6b276cfa14a85a5324728a12b4609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:10 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:10.994750 9944 tablet_replica.cc:333] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:10.994336 9666 tablet_service.cc:1511] Processing CreateTablet for tablet b4dca0cadd5e400eab1365c03a8de4f4 (DEFAULT_TABLE table=default.Table_15bf9089e68d4882838fe4f1c65156ef [id=3814e135130a4213a46df3cd93eca49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:10.996085 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4dca0cadd5e400eab1365c03a8de4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.000160 9516 tablet_service.cc:1558] Processing DeleteTablet for tablet c6a6b276cfa14a85a5324728a12b4609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:10 UTC) from {username='slave'} at 127.0.0.1:52642
W20260504 13:56:10.989259 9339 rpcz_store.cc:269] 0504 13:56:09.968434 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:09.968552 (+ 118us) service_pool.cc:224] Handling call
0504 13:56:09.978653 (+ 10101us) spinlock_profiling.cc:243] Waited 8.83 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:10.988835 (+1010182us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:10.083527 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:10.084356 (+ 829us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:10.084413 (+ 57us) write_op.cc:435] Acquired schema lock
0504 13:56:10.084429 (+ 16us) tablet.cc:662] Decoding operations
0504 13:56:10.084941 (+ 512us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:10.085010 (+ 69us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:10.085068 (+ 58us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:10.085338 (+ 270us) tablet.cc:701] Row locks acquired
0504 13:56:10.085384 (+ 46us) write_op.cc:260] PREPARE: finished
0504 13:56:10.085595 (+ 211us) write_op.cc:270] Start()
0504 13:56:10.085743 (+ 148us) write_op.cc:276] Timestamp: P: 1777902970085511 usec, L: 0
0504 13:56:10.085788 (+ 45us) op_driver.cc:348] REPLICATION: starting
0504 13:56:10.087004 (+ 1216us) log.cc:844] Serialized 532 byte log entry
0504 13:56:10.109811 (+ 22807us) op_driver.cc:464] REPLICATION: finished
0504 13:56:10.110331 (+ 520us) write_op.cc:301] APPLY: starting
0504 13:56:10.110403 (+ 72us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:10.110515 (+ 112us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:10.110646 (+ 131us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:10.110819 (+ 173us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:10.110994 (+ 175us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:10.111060 (+ 66us) write_op.cc:312] APPLY: finished
0504 13:56:10.111255 (+ 195us) log.cc:844] Serialized 22 byte log entry
0504 13:56:10.111388 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:10.111444 (+ 56us) write_op.cc:454] Released schema lock
0504 13:56:10.111472 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:10.166519 (+ 55047us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:10.166553 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":20317568,"child_traces":[["op",{"apply.queue_time_us":349,"apply.run_cpu_time_us":1310,"apply.run_wall_time_us":1310,"num_ops":1,"prepare.queue_time_us":92749,"prepare.run_cpu_time_us":2769,"prepare.run_wall_time_us":82993,"raft.queue_time_us":2613,"raft.run_cpu_time_us":598,"raft.run_wall_time_us":15662,"replication_time_us":23908}]]}
I20260504 13:56:11.001857 9517 tablet_service.cc:1511] Processing CreateTablet for tablet b4dca0cadd5e400eab1365c03a8de4f4 (DEFAULT_TABLE table=default.Table_15bf9089e68d4882838fe4f1c65156ef [id=3814e135130a4213a46df3cd93eca49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.002595 9685 raft_consensus.cc:2468] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:11.003345 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4dca0cadd5e400eab1365c03a8de4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.004159 10002 tablet_replica.cc:333] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:11.005210 10002 raft_consensus.cc:2243] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:11.006044 10002 pending_rounds.cc:70] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00: Trying to abort 1 pending ops.
I20260504 13:56:11.006265 9813 tablet_service.cc:1511] Processing CreateTablet for tablet b4dca0cadd5e400eab1365c03a8de4f4 (DEFAULT_TABLE table=default.Table_15bf9089e68d4882838fe4f1c65156ef [id=3814e135130a4213a46df3cd93eca49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.011880 9814 tablet_service.cc:1511] Processing CreateTablet for tablet 67f26d71286c4f90abd681f9fc9d1e30 (DEFAULT_TABLE table=default.Table_895e961d6c1a45e688ad4e0ca752bae9 [id=73e12524eb9e416782bf9f7fb58c93bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.012759 9664 tablet_service.cc:1511] Processing CreateTablet for tablet 67f26d71286c4f90abd681f9fc9d1e30 (DEFAULT_TABLE table=default.Table_895e961d6c1a45e688ad4e0ca752bae9 [id=73e12524eb9e416782bf9f7fb58c93bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.013744 9664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67f26d71286c4f90abd681f9fc9d1e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.014379 9342 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_eb48d03e69cd40e69ca8ae3e9f3e9dcf" } modify_external_catalogs: true
I20260504 13:56:11.014443 9931 tablet_bootstrap.cc:654] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:11.015873 9335 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_28052c777ee944b79b802d28b415c24e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:11.017551 9335 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28052c777ee944b79b802d28b415c24e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:11.014933 9342 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_eb48d03e69cd40e69ca8ae3e9f3e9dcf" } modify_external_catalogs: true
I20260504 13:56:11.028239 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ff55bef5e54353ab9ca09130ab1e0e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:11.028856 9833 raft_consensus.cc:2468] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:11.032367 9931 tablet_bootstrap.cc:492] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:11.032910 9931 ts_tablet_manager.cc:1403] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.081s user 0.011s sys 0.004s
I20260504 13:56:11.033138 9620 leader_election.cc:304] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:11.033907 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67f26d71286c4f90abd681f9fc9d1e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.034081 9941 raft_consensus.cc:2804] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:11.034418 9941 raft_consensus.cc:493] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:11.037066 9931 raft_consensus.cc:359] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.037989 9931 raft_consensus.cc:385] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:11.038403 9931 raft_consensus.cc:740] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:11.039426 9931 consensus_queue.cc:260] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.045713 9931 ts_tablet_manager.cc:1434] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260504 13:56:11.047089 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4dca0cadd5e400eab1365c03a8de4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.051191 10013 tablet_bootstrap.cc:492] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:11.052517 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 45576.2us
I20260504 13:56:11.006381 10002 pending_rounds.cc:77] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7282290568714567680 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "c6a6b276cfa14a85a5324728a12b4609" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 3 new_table_name: "default.Table_7ac44928fca54f5eaadf591188fc5a03" dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" new_extra_config { } }
I20260504 13:56:11.059451 9968 leader_election.cc:290] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:11.060434 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "385cc1f6f4b24c268c25c61a4741480d" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:11.060961 9833 raft_consensus.cc:3060] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:11.062551 9931 tablet_bootstrap.cc:492] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:11.067517 9833 raft_consensus.cc:2468] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:11.060120 9473 leader_election.cc:304] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:11.067983 9931 tablet_bootstrap.cc:654] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:11.069756 9520 tablet_service.cc:1511] Processing CreateTablet for tablet 67f26d71286c4f90abd681f9fc9d1e30 (DEFAULT_TABLE table=default.Table_895e961d6c1a45e688ad4e0ca752bae9 [id=73e12524eb9e416782bf9f7fb58c93bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.070401 9968 raft_consensus.cc:2804] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:11.070988 9968 raft_consensus.cc:697] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:11.071861 9968 consensus_queue.cc:237] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:11.057974 9516 tablet_service.cc:1511] Processing CreateTablet for tablet eb7f58645c76441589368072274ebc8f (DEFAULT_TABLE table=default.Table_d5cc86d81fb043319576af0a39c643a8 [id=7af4adcf5ee44bad89afa2e4ded416fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.081022 9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb7f58645c76441589368072274ebc8f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:11.054592 10002 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52642: Aborted: Op aborted
I20260504 13:56:11.092449 10002 raft_consensus.cc:2272] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:11.096087 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67f26d71286c4f90abd681f9fc9d1e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.103080 9813 tablet_service.cc:1511] Processing CreateTablet for tablet eb7f58645c76441589368072274ebc8f (DEFAULT_TABLE table=default.Table_d5cc86d81fb043319576af0a39c643a8 [id=7af4adcf5ee44bad89afa2e4ded416fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.104216 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb7f58645c76441589368072274ebc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.107385 9941 raft_consensus.cc:3060] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:11.108037 9338 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_4bdb8491ef7e4ae487fc734f646abf1f" } new_table_name: "default.Table_078d1859743841e985b4e0e1c4f61030" modify_external_catalogs: true
I20260504 13:56:11.115585 9665 tablet_service.cc:1511] Processing CreateTablet for tablet eb7f58645c76441589368072274ebc8f (DEFAULT_TABLE table=default.Table_d5cc86d81fb043319576af0a39c643a8 [id=7af4adcf5ee44bad89afa2e4ded416fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.118376 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_c3d7868e371a4a48b12afd3f38073070"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:11.120023 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3d7868e371a4a48b12afd3f38073070 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:11.120000 9515 tablet_service.cc:1511] Processing CreateTablet for tablet 44017e2079be49258bab49325e3da1c1 (DEFAULT_TABLE table=default.Table_c9e39aac400445028318ffebeb95209d [id=85b12280c48642b2a9cc8c51af037f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.122023 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44017e2079be49258bab49325e3da1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.124239 9663 tablet_service.cc:1511] Processing CreateTablet for tablet 44017e2079be49258bab49325e3da1c1 (DEFAULT_TABLE table=default.Table_c9e39aac400445028318ffebeb95209d [id=85b12280c48642b2a9cc8c51af037f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.126331 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44017e2079be49258bab49325e3da1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.128410 9343 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_39d77929b2fb4ca585977d4c8a362129"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:11.129418 9812 tablet_service.cc:1511] Processing CreateTablet for tablet 44017e2079be49258bab49325e3da1c1 (DEFAULT_TABLE table=default.Table_c9e39aac400445028318ffebeb95209d [id=85b12280c48642b2a9cc8c51af037f2e]), partition=RANGE (key) PARTITION UNBOUNDED
W20260504 13:56:11.129997 9343 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39d77929b2fb4ca585977d4c8a362129 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:11.130523 9812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44017e2079be49258bab49325e3da1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.139514 9342 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet ddd9d68a1e9d45c9b6f9c8a90912ef54
I20260504 13:56:11.142969 9662 tablet_service.cc:1558] Processing DeleteTablet for tablet ddd9d68a1e9d45c9b6f9c8a90912ef54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:11.147388 9662 tablet_service.cc:1511] Processing CreateTablet for tablet 6f31ed9ad6d147d486c44606e68de22e (DEFAULT_TABLE table=default.Table_7b3c23577d974ef0bdf80b361e3738c6 [id=df5f0bd632b2428492a2954070269840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.148132 9514 tablet_service.cc:1558] Processing DeleteTablet for tablet ddd9d68a1e9d45c9b6f9c8a90912ef54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:11.148582 9811 tablet_service.cc:1558] Processing DeleteTablet for tablet ddd9d68a1e9d45c9b6f9c8a90912ef54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:11.144837 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb7f58645c76441589368072274ebc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.149634 9513 tablet_service.cc:1511] Processing CreateTablet for tablet 6f31ed9ad6d147d486c44606e68de22e (DEFAULT_TABLE table=default.Table_7b3c23577d974ef0bdf80b361e3738c6 [id=df5f0bd632b2428492a2954070269840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.150820 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f31ed9ad6d147d486c44606e68de22e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.154693 9342 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_c9c66820c2c24a1795b44da8856630d7" } new_table_name: "default.Table_a3c5361bede04e74b57b95480a479298" modify_external_catalogs: true
I20260504 13:56:11.157593 10003 ts_tablet_manager.cc:1929] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:11.160522 9662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f31ed9ad6d147d486c44606e68de22e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.162679 9344 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 39f080a0296b44988e8e51e68453a551 (table default.Table_00c39fd2584d48fb965f7e81908e2ae7 [id=532ece9a0c134c1f8c1c3793638f34c5]). Expected version 1 got 0
I20260504 13:56:11.165712 9514 tablet_service.cc:1511] Processing CreateTablet for tablet 987f77386b504f17ab62c796f78f4cb2 (DEFAULT_TABLE table=default.Table_c9c66820c2c24a1795b44da8856630d7 [id=6949858f6acf4cec88b9af9791e5c693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.166923 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 987f77386b504f17ab62c796f78f4cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.160370 9811 tablet_service.cc:1511] Processing CreateTablet for tablet 6f31ed9ad6d147d486c44606e68de22e (DEFAULT_TABLE table=default.Table_7b3c23577d974ef0bdf80b361e3738c6 [id=df5f0bd632b2428492a2954070269840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.172950 9811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f31ed9ad6d147d486c44606e68de22e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.188249 9947 raft_consensus.cc:515] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:11.190037 9947 leader_election.cc:290] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:11.191767 10002 ts_tablet_manager.cc:1916] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:11.202786 10003 log.cc:1199] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/8d25c16864f343a88fa77535831d5a52
I20260504 13:56:11.203562 10003 ts_tablet_manager.cc:1950] T 8d25c16864f343a88fa77535831d5a52 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:11.203471 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ff55bef5e54353ab9ca09130ab1e0e" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:11.203998 9833 raft_consensus.cc:3060] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:11.180029 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_c15418510612433e86cd370f162b6c31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:11.188262 9335 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_512c72ff320944448051ca113741e6a4" } modify_external_catalogs: true
I20260504 13:56:11.213285 9335 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_512c72ff320944448051ca113741e6a4" } modify_external_catalogs: true
I20260504 13:56:11.206128 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 8d25c16864f343a88fa77535831d5a52 (table default.Table_c943f49871964cdc9c802e13bfb75928 [id=ed4c94257e234052adc4d311758ebbd3]) successfully deleted
W20260504 13:56:11.211555 9338 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03818e34ac0545be880c873cf82cd4ea has no consensus state
I20260504 13:56:11.219903 9814 tablet_service.cc:1511] Processing CreateTablet for tablet 987f77386b504f17ab62c796f78f4cb2 (DEFAULT_TABLE table=default.Table_c9c66820c2c24a1795b44da8856630d7 [id=6949858f6acf4cec88b9af9791e5c693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.221031 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 987f77386b504f17ab62c796f78f4cb2. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:11.222676 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c15418510612433e86cd370f162b6c31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:11.228478 10003 tablet_replica.cc:333] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:11.229292 10003 raft_consensus.cc:2243] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:11.229846 10003 pending_rounds.cc:70] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888: Trying to abort 1 pending ops.
I20260504 13:56:11.230142 10003 pending_rounds.cc:77] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7282290568714567680 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "c6a6b276cfa14a85a5324728a12b4609" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 3 new_table_name: "default.Table_7ac44928fca54f5eaadf591188fc5a03" dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" new_extra_config { } }
I20260504 13:56:11.249516 10013 tablet_bootstrap.cc:654] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:11.256945 10003 raft_consensus.cc:2272] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:11.259750 10003 ts_tablet_manager.cc:1916] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260504 13:56:11.273056 9342 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 987f77386b504f17ab62c796f78f4cb2 consensus state has no leader
I20260504 13:56:11.278260 9931 tablet_bootstrap.cc:492] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:11.283569 9931 ts_tablet_manager.cc:1403] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.221s user 0.034s sys 0.009s
I20260504 13:56:11.286499 9931 raft_consensus.cc:359] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.293561 9931 raft_consensus.cc:385] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:11.293887 9931 raft_consensus.cc:740] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:11.294538 9931 consensus_queue.cc:260] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.291631 9833 raft_consensus.cc:2468] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:11.296154 9474 leader_election.cc:304] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:11.297017 9947 raft_consensus.cc:2804] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:11.298025 9947 raft_consensus.cc:697] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:11.300237 9947 consensus_queue.cc:237] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:11.297948 9661 tablet_service.cc:1511] Processing CreateTablet for tablet 987f77386b504f17ab62c796f78f4cb2 (DEFAULT_TABLE table=default.Table_c9c66820c2c24a1795b44da8856630d7 [id=6949858f6acf4cec88b9af9791e5c693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.300966 9932 tablet_bootstrap.cc:492] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:11.301335 9932 ts_tablet_manager.cc:1403] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.388s user 0.020s sys 0.005s
I20260504 13:56:11.306567 9661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 987f77386b504f17ab62c796f78f4cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.309432 9938 raft_consensus.cc:493] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:11.309845 9938 raft_consensus.cc:515] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.311733 9938 leader_election.cc:290] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:11.313308 9335 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet dcc02d5f590e420987daab92fdf8b10a
I20260504 13:56:11.318660 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4dca0cadd5e400eab1365c03a8de4f4" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:11.319993 9767 leader_election.cc:343] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:11.323027 9931 ts_tablet_manager.cc:1434] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.039s user 0.004s sys 0.001s
I20260504 13:56:11.326944 9931 tablet_bootstrap.cc:492] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:11.335964 9328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 44586.2us
I20260504 13:56:11.348449 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ff55bef5e54353ab9ca09130ab1e0e" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:11.354260 9944 raft_consensus.cc:2243] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:11.356891 9944 pending_rounds.cc:70] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6: Trying to abort 1 pending ops.
W20260504 13:56:11.357064 9328 consensus_peers.cc:597] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 -> Peer 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337): Couldn't send request to peer 6caedf407d13459e8fd477e8b60b7dc7. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260504 13:56:11.357162 9944 pending_rounds.cc:77] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7282290568714567680 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "c6a6b276cfa14a85a5324728a12b4609" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 3 new_table_name: "default.Table_7ac44928fca54f5eaadf591188fc5a03" dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" new_extra_config { } }
I20260504 13:56:11.359306 9944 raft_consensus.cc:2272] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:11.360870 9344 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_c3d7868e371a4a48b12afd3f38073070" } modify_external_catalogs: true
I20260504 13:56:11.361444 9931 tablet_bootstrap.cc:654] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:11.361320 9344 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_c3d7868e371a4a48b12afd3f38073070" } modify_external_catalogs: true
I20260504 13:56:11.362776 9944 ts_tablet_manager.cc:1916] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:11.368101 9941 raft_consensus.cc:515] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } }
I20260504 13:56:11.369246 9685 raft_consensus.cc:2393] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 1: Already voted for candidate 5beceb6317444247b31b44a6581707c6 in this term.
I20260504 13:56:11.372666 9941 leader_election.cc:290] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:11.373658 10003 ts_tablet_manager.cc:1929] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260504 13:56:11.374065 10003 log.cc:1199] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/c6a6b276cfa14a85a5324728a12b4609
I20260504 13:56:11.374820 10003 ts_tablet_manager.cc:1950] T c6a6b276cfa14a85a5324728a12b4609 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:11.376389 10002 ts_tablet_manager.cc:1929] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260504 13:56:11.376780 10002 log.cc:1199] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/c6a6b276cfa14a85a5324728a12b4609
I20260504 13:56:11.378223 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet c6a6b276cfa14a85a5324728a12b4609 (table default.Table_7ac44928fca54f5eaadf591188fc5a03 [id=15c00efe5d7e4f2586f9f0dab975346f]) successfully deleted
I20260504 13:56:11.382375 9344 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet f27aa75d776b4831a93ffc859ae8022c
I20260504 13:56:11.386906 10003 tablet_replica.cc:333] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:11.387327 9344 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_49715b69b840473b9f7b0fdabcc0084e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:11.387760 10003 raft_consensus.cc:2243] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:11.388207 10003 raft_consensus.cc:2272] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:11.389017 9344 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49715b69b840473b9f7b0fdabcc0084e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:11.353601 10004 tablet.cc:1724] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260504 13:56:11.400475 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ff55bef5e54353ab9ca09130ab1e0e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:11.401199 9833 raft_consensus.cc:2393] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5beceb6317444247b31b44a6581707c6 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:11.405113 10003 ts_tablet_manager.cc:1916] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:11.406069 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ff55bef5e54353ab9ca09130ab1e0e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:11.407457 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ff55bef5e54353ab9ca09130ab1e0e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:11.407498 9932 raft_consensus.cc:359] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.408066 9932 raft_consensus.cc:385] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:11.408305 9932 raft_consensus.cc:740] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:11.408864 9932 consensus_queue.cc:260] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.410203 9618 leader_election.cc:304] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:11.422840 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4dca0cadd5e400eab1365c03a8de4f4" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
W20260504 13:56:11.425964 9768 leader_election.cc:343] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:11.426384 9768 leader_election.cc:304] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:11.426424 9343 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_7e28164fb19d47649ad4454da7cc5f85" } new_table_name: "default.Table_4cf6527a29704312924d2bf06660d6d9" modify_external_catalogs: true
I20260504 13:56:11.427403 9938 raft_consensus.cc:2749] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:11.429656 10002 ts_tablet_manager.cc:1950] T c6a6b276cfa14a85a5324728a12b4609 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:11.430807 10013 tablet_bootstrap.cc:492] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:11.431321 10013 ts_tablet_manager.cc:1403] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.380s user 0.020s sys 0.002s
I20260504 13:56:11.432485 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet c6a6b276cfa14a85a5324728a12b4609 (table default.Table_7ac44928fca54f5eaadf591188fc5a03 [id=15c00efe5d7e4f2586f9f0dab975346f]) successfully deleted
W20260504 13:56:11.433565 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet ddd9d68a1e9d45c9b6f9c8a90912ef54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddd9d68a1e9d45c9b6f9c8a90912ef54 already in progress: creating tablet
I20260504 13:56:11.436079 10013 raft_consensus.cc:359] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.441056 10013 raft_consensus.cc:385] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:11.440330 9335 catalog_manager.cc:5671] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:11.441401 10013 raft_consensus.cc:740] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:11.442507 9931 tablet_bootstrap.cc:492] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:11.442098 10013 consensus_queue.cc:260] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.442893 9931 ts_tablet_manager.cc:1403] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.116s user 0.017s sys 0.004s
I20260504 13:56:11.442399 9335 catalog_manager.cc:5671] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:11.443658 9941 raft_consensus.cc:2749] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260504 13:56:11.443925 9335 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 5625163f6add4a20b1828905d90b5486 (table default.Table_56891f8543a84e859c39b381625a9daa [id=7b3ef250728944d9998ef2df032f8819]). Expected version 1 got 0
I20260504 13:56:11.444823 9335 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 (table default.Table_02ca7067550b4045804cc5764e09c086 [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 1 got 0
I20260504 13:56:11.446609 9931 raft_consensus.cc:359] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.447225 9931 raft_consensus.cc:385] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:11.447503 9931 raft_consensus.cc:740] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:11.446090 9339 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet b4dca0cadd5e400eab1365c03a8de4f4 (table default.Table_15bf9089e68d4882838fe4f1c65156ef [id=3814e135130a4213a46df3cd93eca49e]). Expected version 1 got 0
I20260504 13:56:11.451550 9931 consensus_queue.cc:260] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.453022 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_7062b77a3b7d42678cbb8c4a6134be59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:11.454599 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7062b77a3b7d42678cbb8c4a6134be59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:11.457629 9342 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet b4dca0cadd5e400eab1365c03a8de4f4 (table default.Table_15bf9089e68d4882838fe4f1c65156ef [id=3814e135130a4213a46df3cd93eca49e]). Expected version 1 got 0
I20260504 13:56:11.458636 9931 ts_tablet_manager.cc:1434] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20260504 13:56:11.461230 10003 ts_tablet_manager.cc:1929] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:11.461613 10003 log.cc:1199] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/ddd9d68a1e9d45c9b6f9c8a90912ef54
I20260504 13:56:11.462280 10003 ts_tablet_manager.cc:1950] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:11.465225 10015 tablet.cc:1719] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6: Skipping requested alter to schema version 3, tablet already version 3
I20260504 13:56:11.467413 9932 ts_tablet_manager.cc:1434] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.166s user 0.006s sys 0.001s
I20260504 13:56:11.468189 9932 tablet_bootstrap.cc:492] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:11.468032 9515 tablet_service.cc:1511] Processing CreateTablet for tablet b320a37b29844c7593a12000eaa3da43 (DEFAULT_TABLE table=default.Table_2099388b6e2a455295be76a6d072758e [id=63e34a66cd644ab7bede27373e321714]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.468827 10013 ts_tablet_manager.cc:1434] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.037s user 0.004s sys 0.001s
I20260504 13:56:11.469125 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b320a37b29844c7593a12000eaa3da43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.469682 10013 tablet_bootstrap.cc:492] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:11.470445 9931 tablet_bootstrap.cc:492] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:11.478386 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet ddd9d68a1e9d45c9b6f9c8a90912ef54 (table default.Table_eb48d03e69cd40e69ca8ae3e9f3e9dcf [id=0d9014419f714d2c82f3a861926fa0c8]) successfully deleted
W20260504 13:56:11.481614 9618 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 71597.6us
I20260504 13:56:11.482755 9931 tablet_bootstrap.cc:654] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:11.496538 9931 tablet_bootstrap.cc:492] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:11.496914 9931 ts_tablet_manager.cc:1403] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.000s
I20260504 13:56:11.497428 9337 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_02ca7067550b4045804cc5764e09c086" } new_table_name: "default.Table_0eae04160cf241df95cc53da6f4dfacf" modify_external_catalogs: true
I20260504 13:56:11.499234 9931 raft_consensus.cc:359] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.499833 9931 raft_consensus.cc:385] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:11.500087 9931 raft_consensus.cc:740] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:11.501443 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_ea18464634544273a0bbb10fce8b88ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:11.503006 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea18464634544273a0bbb10fce8b88ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:11.502938 9931 consensus_queue.cc:260] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.508249 9931 ts_tablet_manager.cc:1434] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.011s user 0.004s sys 0.003s
I20260504 13:56:11.514051 9944 ts_tablet_manager.cc:1929] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260504 13:56:11.514484 9944 log.cc:1199] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/c6a6b276cfa14a85a5324728a12b4609
I20260504 13:56:11.516975 9931 tablet_bootstrap.cc:492] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:11.519788 9932 tablet_bootstrap.cc:654] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:11.519788 10013 tablet_bootstrap.cc:654] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:11.521991 9931 tablet_bootstrap.cc:654] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:11.522536 9343 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b7a71dbcc694e1a97019188cfb2c672 consensus state has no leader
I20260504 13:56:11.527341 9662 tablet_service.cc:1511] Processing CreateTablet for tablet b320a37b29844c7593a12000eaa3da43 (DEFAULT_TABLE table=default.Table_2099388b6e2a455295be76a6d072758e [id=63e34a66cd644ab7bede27373e321714]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.527697 9814 tablet_service.cc:1511] Processing CreateTablet for tablet b320a37b29844c7593a12000eaa3da43 (DEFAULT_TABLE table=default.Table_2099388b6e2a455295be76a6d072758e [id=63e34a66cd644ab7bede27373e321714]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.528477 9662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b320a37b29844c7593a12000eaa3da43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.528908 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b320a37b29844c7593a12000eaa3da43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.483271 9335 catalog_manager.cc:5671] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:11.529117 9520 tablet_service.cc:1511] Processing CreateTablet for tablet 03818e34ac0545be880c873cf82cd4ea (DEFAULT_TABLE table=default.Table_078d1859743841e985b4e0e1c4f61030 [id=89a6d18328c04d90828f3e51696592fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.529608 9335 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 385cc1f6f4b24c268c25c61a4741480d (table default.Table_659330a3d91d404ebc0d526364ed2f0b [id=0b1606cb64b440cea0b713bd369c440d]). Expected version 1 got 0
I20260504 13:56:11.530151 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03818e34ac0545be880c873cf82cd4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.534765 9938 raft_consensus.cc:493] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:11.535146 9938 raft_consensus.cc:515] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.536903 9938 leader_election.cc:290] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:11.538205 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39f080a0296b44988e8e51e68453a551" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:11.538666 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39f080a0296b44988e8e51e68453a551" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:11.539273 9540 raft_consensus.cc:2468] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
W20260504 13:56:11.539817 9768 leader_election.cc:343] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:11.541214 9767 leader_election.cc:304] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:11.542353 9938 raft_consensus.cc:2804] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:11.542680 9938 raft_consensus.cc:493] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:11.542560 9339 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_659330a3d91d404ebc0d526364ed2f0b" } modify_external_catalogs: true
I20260504 13:56:11.542974 9938 raft_consensus.cc:3060] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:11.543088 9339 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_659330a3d91d404ebc0d526364ed2f0b" } modify_external_catalogs: true
I20260504 13:56:11.543895 9944 ts_tablet_manager.cc:1950] T c6a6b276cfa14a85a5324728a12b4609 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:11.546586 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet c6a6b276cfa14a85a5324728a12b4609 (table default.Table_7ac44928fca54f5eaadf591188fc5a03 [id=15c00efe5d7e4f2586f9f0dab975346f]) successfully deleted
I20260504 13:56:11.549414 9944 tablet_replica.cc:333] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:11.550201 9944 raft_consensus.cc:2243] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:11.550691 9944 raft_consensus.cc:2272] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:11.549223 9938 raft_consensus.cc:515] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.555701 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_e282b7831dc24295974e99bb900ca585"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:11.557307 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e282b7831dc24295974e99bb900ca585 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:11.560401 10013 tablet_bootstrap.cc:492] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:11.560810 10013 ts_tablet_manager.cc:1403] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.091s user 0.011s sys 0.000s
I20260504 13:56:11.561447 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39f080a0296b44988e8e51e68453a551" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
W20260504 13:56:11.562588 9768 leader_election.cc:343] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:11.563364 10013 raft_consensus.cc:359] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.564262 10013 raft_consensus.cc:385] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:11.564518 10013 raft_consensus.cc:740] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:11.564781 9932 tablet_bootstrap.cc:492] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:11.565136 9932 ts_tablet_manager.cc:1403] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.005s
I20260504 13:56:11.565382 10013 consensus_queue.cc:260] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.566180 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "39f080a0296b44988e8e51e68453a551" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:11.566673 9540 raft_consensus.cc:3060] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:11.571471 9932 raft_consensus.cc:359] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.572043 9932 raft_consensus.cc:385] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:11.572278 10013 ts_tablet_manager.cc:1434] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20260504 13:56:11.572285 9932 raft_consensus.cc:740] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:11.573141 10013 tablet_bootstrap.cc:492] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:11.572876 9932 consensus_queue.cc:260] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.575016 9932 ts_tablet_manager.cc:1434] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260504 13:56:11.588598 9540 raft_consensus.cc:2468] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:11.591907 9767 leader_election.cc:304] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:11.596745 9944 ts_tablet_manager.cc:1916] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:11.600834 9341 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_078d1859743841e985b4e0e1c4f61030" } modify_external_catalogs: true
I20260504 13:56:11.601399 9341 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_078d1859743841e985b4e0e1c4f61030" } modify_external_catalogs: true
I20260504 13:56:11.603374 9932 tablet_bootstrap.cc:492] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:11.608945 9932 tablet_bootstrap.cc:654] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:11.611528 9938 leader_election.cc:290] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:11.613488 9661 tablet_service.cc:1511] Processing CreateTablet for tablet 03818e34ac0545be880c873cf82cd4ea (DEFAULT_TABLE table=default.Table_078d1859743841e985b4e0e1c4f61030 [id=89a6d18328c04d90828f3e51696592fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.613932 9684 raft_consensus.cc:1275] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Refusing update from remote peer d4051d2bf0bf410eb8acd107560df888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 13:56:11.614534 9661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03818e34ac0545be880c873cf82cd4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.619760 9938 raft_consensus.cc:2804] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:11.621194 9938 raft_consensus.cc:697] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:11.622970 9938 consensus_queue.cc:237] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.623502 10013 tablet_bootstrap.cc:654] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:11.625841 9520 tablet_service.cc:1511] Processing CreateTablet for tablet 0b7a71dbcc694e1a97019188cfb2c672 (DEFAULT_TABLE table=default.Table_4cf6527a29704312924d2bf06660d6d9 [id=0649fc4627314bca8b51d0e1f5022e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.625690 9343 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_b97e5fa65d9e40c9982c796a6ca8afe1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:11.626888 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b7a71dbcc694e1a97019188cfb2c672. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:11.628006 9343 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b97e5fa65d9e40c9982c796a6ca8afe1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:11.624897 10032 consensus_queue.cc:1048] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:11.626791 9811 tablet_service.cc:1511] Processing CreateTablet for tablet 03818e34ac0545be880c873cf82cd4ea (DEFAULT_TABLE table=default.Table_078d1859743841e985b4e0e1c4f61030 [id=89a6d18328c04d90828f3e51696592fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.635931 9811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03818e34ac0545be880c873cf82cd4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.647541 10013 tablet_bootstrap.cc:492] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:11.647948 10013 ts_tablet_manager.cc:1403] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.075s user 0.011s sys 0.000s
I20260504 13:56:11.650497 10013 raft_consensus.cc:359] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.651114 10013 raft_consensus.cc:385] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:11.651396 10013 raft_consensus.cc:740] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:11.655239 9540 raft_consensus.cc:1275] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Refusing update from remote peer d4051d2bf0bf410eb8acd107560df888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 13:56:11.661813 10013 consensus_queue.cc:260] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.666304 9944 ts_tablet_manager.cc:1929] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:11.666680 9944 log.cc:1199] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/ddd9d68a1e9d45c9b6f9c8a90912ef54
I20260504 13:56:11.666287 10032 consensus_queue.cc:1048] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260504 13:56:11.677390 10013 ts_tablet_manager.cc:1434] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20260504 13:56:11.680907 10013 tablet_bootstrap.cc:492] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:11.680989 9931 tablet_bootstrap.cc:492] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:11.681408 9931 ts_tablet_manager.cc:1403] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.165s user 0.004s sys 0.004s
I20260504 13:56:11.685453 9947 consensus_queue.cc:1048] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:11.686204 9662 tablet_service.cc:1511] Processing CreateTablet for tablet 0b7a71dbcc694e1a97019188cfb2c672 (DEFAULT_TABLE table=default.Table_4cf6527a29704312924d2bf06660d6d9 [id=0649fc4627314bca8b51d0e1f5022e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.688203 9814 tablet_service.cc:1511] Processing CreateTablet for tablet 0b7a71dbcc694e1a97019188cfb2c672 (DEFAULT_TABLE table=default.Table_4cf6527a29704312924d2bf06660d6d9 [id=0649fc4627314bca8b51d0e1f5022e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.690318 9520 tablet_service.cc:1511] Processing CreateTablet for tablet 91371bf848f5465ea2db0c090583382d (DEFAULT_TABLE table=default.Table_28052c777ee944b79b802d28b415c24e [id=994b65b95f66440283f852cd9dd9ae0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.690507 9931 raft_consensus.cc:359] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.691145 9931 raft_consensus.cc:385] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:11.691429 9931 raft_consensus.cc:740] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:11.691725 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91371bf848f5465ea2db0c090583382d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.692078 9931 consensus_queue.cc:260] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.693998 9931 ts_tablet_manager.cc:1434] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.012s user 0.002s sys 0.004s
I20260504 13:56:11.672961 9341 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 03818e34ac0545be880c873cf82cd4ea
I20260504 13:56:11.696259 9663 tablet_service.cc:1511] Processing CreateTablet for tablet 91371bf848f5465ea2db0c090583382d (DEFAULT_TABLE table=default.Table_28052c777ee944b79b802d28b415c24e [id=994b65b95f66440283f852cd9dd9ae0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.697739 9514 tablet_service.cc:1558] Processing DeleteTablet for tablet 03818e34ac0545be880c873cf82cd4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:11.698050 9665 tablet_service.cc:1558] Processing DeleteTablet for tablet 03818e34ac0545be880c873cf82cd4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:11.699339 9662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b7a71dbcc694e1a97019188cfb2c672. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:11.699679 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 03818e34ac0545be880c873cf82cd4ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03818e34ac0545be880c873cf82cd4ea already in progress: creating tablet
I20260504 13:56:11.700089 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91371bf848f5465ea2db0c090583382d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.703364 9947 raft_consensus.cc:493] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:11.703755 9947 raft_consensus.cc:515] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.704277 9813 tablet_service.cc:1558] Processing DeleteTablet for tablet 03818e34ac0545be880c873cf82cd4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:11.705549 9947 leader_election.cc:290] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:11.709048 9338 catalog_manager.cc:5671] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:11.711133 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddd9d68a1e9d45c9b6f9c8a90912ef54" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
W20260504 13:56:11.712152 9473 leader_election.cc:343] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: Tablet not RUNNING: STOPPED
I20260504 13:56:11.712432 9338 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 39f080a0296b44988e8e51e68453a551 (table default.Table_00c39fd2584d48fb965f7e81908e2ae7 [id=532ece9a0c134c1f8c1c3793638f34c5]). Expected version 1 got 0
I20260504 13:56:11.715884 10013 tablet_bootstrap.cc:654] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:11.717530 9474 leader_election.cc:343] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Not found: Tablet not found: ddd9d68a1e9d45c9b6f9c8a90912ef54
I20260504 13:56:11.717918 9474 leader_election.cc:304] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:11.719508 9932 tablet_bootstrap.cc:492] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:11.719509 9947 raft_consensus.cc:2749] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:11.719875 9932 ts_tablet_manager.cc:1403] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.130s user 0.002s sys 0.011s
I20260504 13:56:11.719887 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b7a71dbcc694e1a97019188cfb2c672. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.726694 9341 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_7b3c23577d974ef0bdf80b361e3738c6" } new_table_name: "default.Table_e864e83fbba04cc3abbf7a6e955ac912" modify_external_catalogs: true
W20260504 13:56:11.731858 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 03818e34ac0545be880c873cf82cd4ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03818e34ac0545be880c873cf82cd4ea already in progress: creating tablet
I20260504 13:56:11.739624 10004 tablet.cc:1724] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:11.743685 9811 tablet_service.cc:1511] Processing CreateTablet for tablet 91371bf848f5465ea2db0c090583382d (DEFAULT_TABLE table=default.Table_28052c777ee944b79b802d28b415c24e [id=994b65b95f66440283f852cd9dd9ae0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:11.744901 9811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91371bf848f5465ea2db0c090583382d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:11.746436 9947 raft_consensus.cc:493] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:11.746862 9947 raft_consensus.cc:515] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.749146 9947 leader_election.cc:290] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
W20260504 13:56:11.751340 9474 leader_election.cc:343] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Not found: Tablet not found: 8d25c16864f343a88fa77535831d5a52
I20260504 13:56:11.760056 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_cdaaea60f9354320bc4cfee6729ba12f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:11.760588 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d25c16864f343a88fa77535831d5a52" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:11.761165 9684 raft_consensus.cc:2468] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
W20260504 13:56:11.761803 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdaaea60f9354320bc4cfee6729ba12f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:11.762840 9473 leader_election.cc:304] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:11.765132 9947 raft_consensus.cc:2804] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:11.765520 9947 raft_consensus.cc:493] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:11.765923 9947 raft_consensus.cc:3060] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:11.766683 9344 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 2951cbd2df4f402eabeef323d121ce1b (table default.Table_d6ee06783219496b94ecc1a88d13b0da [id=46ac1f0baae0492984e87927e4905343]). Expected version 3 got 0
I20260504 13:56:11.767428 9339 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 385cc1f6f4b24c268c25c61a4741480d
I20260504 13:56:11.770768 9520 tablet_service.cc:1558] Processing DeleteTablet for tablet 385cc1f6f4b24c268c25c61a4741480d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:11.771459 10002 tablet_replica.cc:333] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:11.772202 10002 raft_consensus.cc:2243] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:11.772872 10002 pending_rounds.cc:70] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00: Trying to abort 1 pending ops.
I20260504 13:56:11.773090 10002 pending_rounds.cc:77] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290569699045376 op_type: NO_OP noop_request { }
I20260504 13:56:11.773425 10002 raft_consensus.cc:2889] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:11.773795 10002 raft_consensus.cc:2272] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:11.773622 9947 raft_consensus.cc:515] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.775585 9520 tablet_service.cc:1558] Processing DeleteTablet for tablet c6a6b276cfa14a85a5324728a12b4609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:10 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:11.776202 9947 leader_election.cc:290] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
W20260504 13:56:11.777420 9474 leader_election.cc:343] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Not found: Tablet not found: 8d25c16864f343a88fa77535831d5a52
I20260504 13:56:11.779857 10002 ts_tablet_manager.cc:1916] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:11.780300 9661 tablet_service.cc:1558] Processing DeleteTablet for tablet 385cc1f6f4b24c268c25c61a4741480d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:11.780795 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d25c16864f343a88fa77535831d5a52" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:11.781337 9684 raft_consensus.cc:3060] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:11.781401 9813 tablet_service.cc:1558] Processing DeleteTablet for tablet 385cc1f6f4b24c268c25c61a4741480d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:11.782068 10003 tablet_replica.cc:333] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:11.782835 10003 raft_consensus.cc:2243] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:11.783382 10003 raft_consensus.cc:2272] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:11.785148 10003 ts_tablet_manager.cc:1916] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:11.774058 9931 tablet_bootstrap.cc:492] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:11.801354 9661 tablet_service.cc:1558] Processing DeleteTablet for tablet c6a6b276cfa14a85a5324728a12b4609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:10 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:11.776695 9944 ts_tablet_manager.cc:1950] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:11.804996 9932 raft_consensus.cc:359] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.805725 9932 raft_consensus.cc:385] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:11.805984 9932 raft_consensus.cc:740] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:11.806634 9932 consensus_queue.cc:260] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.807348 9684 raft_consensus.cc:2468] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:11.810842 9473 leader_election.cc:304] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:11.821101 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_61add6c7b0dd46b08c0e39ad6d697e8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:11.822650 9947 raft_consensus.cc:2804] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:11.823091 9947 raft_consensus.cc:697] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
W20260504 13:56:11.823112 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61add6c7b0dd46b08c0e39ad6d697e8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:11.823876 9947 consensus_queue.cc:237] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.826550 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet ddd9d68a1e9d45c9b6f9c8a90912ef54 (table default.Table_eb48d03e69cd40e69ca8ae3e9f3e9dcf [id=0d9014419f714d2c82f3a861926fa0c8]) successfully deleted
I20260504 13:56:11.826854 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_fa64d5b0eb084f2a9ab924aa2cdf3a1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:11.828293 9329 catalog_manager.cc:4982] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 03818e34ac0545be880c873cf82cd4ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03818e34ac0545be880c873cf82cd4ea already in progress: creating tablet
I20260504 13:56:11.828658 10041 tablet.cc:1724] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:11.829090 9944 tablet_replica.cc:333] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
W20260504 13:56:11.829123 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa64d5b0eb084f2a9ab924aa2cdf3a1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:11.829828 9944 raft_consensus.cc:2243] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:11.830247 9944 raft_consensus.cc:2272] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:11.831987 9335 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_7062b77a3b7d42678cbb8c4a6134be59" } new_table_name: "default.Table_a0a1002687744b879e47ae4fd701d903" modify_external_catalogs: true
I20260504 13:56:11.808899 9932 ts_tablet_manager.cc:1434] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.089s user 0.005s sys 0.001s
I20260504 13:56:11.847918 9932 tablet_bootstrap.cc:492] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:11.856186 9944 ts_tablet_manager.cc:1916] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:11.857106 9931 tablet_bootstrap.cc:654] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:11.858315 10002 ts_tablet_manager.cc:1929] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:11.859584 10002 log.cc:1199] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/385cc1f6f4b24c268c25c61a4741480d
I20260504 13:56:11.863062 10031 raft_consensus.cc:493] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:11.863926 10002 ts_tablet_manager.cc:1950] T 385cc1f6f4b24c268c25c61a4741480d P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:11.863750 10031 raft_consensus.cc:515] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.866009 10031 leader_election.cc:290] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:11.866622 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67f26d71286c4f90abd681f9fc9d1e30" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
W20260504 13:56:11.866103 9328 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 03818e34ac0545be880c873cf82cd4ea (table default.Table_078d1859743841e985b4e0e1c4f61030 [id=89a6d18328c04d90828f3e51696592fa], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 03818e34ac0545be880c873cf82cd4ea consensus state has no leader
W20260504 13:56:11.868003 9768 leader_election.cc:343] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:11.868258 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 385cc1f6f4b24c268c25c61a4741480d (table default.Table_659330a3d91d404ebc0d526364ed2f0b [id=0b1606cb64b440cea0b713bd369c440d]) successfully deleted
W20260504 13:56:11.869576 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet c6a6b276cfa14a85a5324728a12b4609 because the tablet was not found. No further retry: Not found: Tablet not found: c6a6b276cfa14a85a5324728a12b4609
I20260504 13:56:11.904923 9932 tablet_bootstrap.cc:654] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:11.909552 9344 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 (table default.Table_0eae04160cf241df95cc53da6f4dfacf [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 2 got 0
I20260504 13:56:11.910763 9344 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 39f080a0296b44988e8e51e68453a551 (table default.Table_00c39fd2584d48fb965f7e81908e2ae7 [id=532ece9a0c134c1f8c1c3793638f34c5]). Expected version 1 got 0
I20260504 13:56:11.912822 9338 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_a3c5361bede04e74b57b95480a479298" } modify_external_catalogs: true
I20260504 13:56:11.913473 9338 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_a3c5361bede04e74b57b95480a479298" } modify_external_catalogs: true
I20260504 13:56:11.926095 9515 tablet_service.cc:1558] Processing DeleteTablet for tablet 8d25c16864f343a88fa77535831d5a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:10 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:11.926971 10002 tablet_replica.cc:333] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:11.927765 10002 raft_consensus.cc:2243] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:11.927961 9340 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 5625163f6add4a20b1828905d90b5486 (table default.Table_56891f8543a84e859c39b381625a9daa [id=7b3ef250728944d9998ef2df032f8819]). Expected version 1 got 0
I20260504 13:56:11.928422 10002 pending_rounds.cc:70] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00: Trying to abort 1 pending ops.
I20260504 13:56:11.928659 10002 pending_rounds.cc:77] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290572678672384 op_type: NO_OP noop_request { }
I20260504 13:56:11.928958 10002 raft_consensus.cc:2889] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:11.929265 10002 raft_consensus.cc:2272] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:11.935420 10002 ts_tablet_manager.cc:1916] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:11.940716 9515 tablet_service.cc:1558] Processing DeleteTablet for tablet ddd9d68a1e9d45c9b6f9c8a90912ef54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:11.940891 9932 tablet_bootstrap.cc:492] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:11.941323 9932 ts_tablet_manager.cc:1403] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.094s user 0.009s sys 0.001s
I20260504 13:56:11.943681 9932 raft_consensus.cc:359] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.951366 9932 raft_consensus.cc:385] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:11.955269 9932 raft_consensus.cc:740] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:11.957557 9931 tablet_bootstrap.cc:492] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:11.957955 9931 ts_tablet_manager.cc:1403] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.212s user 0.004s sys 0.005s
I20260504 13:56:11.958017 9932 consensus_queue.cc:260] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:11.958455 9341 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f31ed9ad6d147d486c44606e68de22e consensus state has no leader
I20260504 13:56:11.962375 9932 ts_tablet_manager.cc:1434] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20260504 13:56:11.967370 10013 tablet_bootstrap.cc:492] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:11.968288 9932 tablet_bootstrap.cc:492] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:11.967784 10013 ts_tablet_manager.cc:1403] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.287s user 0.035s sys 0.036s
I20260504 13:56:11.972524 10013 raft_consensus.cc:359] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.973170 10013 raft_consensus.cc:385] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:11.973445 10013 raft_consensus.cc:740] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:11.977212 10041 tablet.cc:1719] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888: Skipping requested alter to schema version 2, tablet already version 2
I20260504 13:56:11.978756 9932 tablet_bootstrap.cc:654] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:11.980856 9344 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_e864e83fbba04cc3abbf7a6e955ac912" } modify_external_catalogs: true
I20260504 13:56:11.980937 10043 consensus_queue.cc:1048] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:11.981350 9344 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_e864e83fbba04cc3abbf7a6e955ac912" } modify_external_catalogs: true
I20260504 13:56:11.979740 10013 consensus_queue.cc:260] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.992503 9931 raft_consensus.cc:359] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.992905 10003 ts_tablet_manager.cc:1929] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:11.993145 9931 raft_consensus.cc:385] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:11.993265 10003 log.cc:1199] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/385cc1f6f4b24c268c25c61a4741480d
I20260504 13:56:11.993414 9931 raft_consensus.cc:740] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:11.993968 10003 ts_tablet_manager.cc:1950] T 385cc1f6f4b24c268c25c61a4741480d P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:11.994079 9931 consensus_queue.cc:260] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:11.997030 9931 ts_tablet_manager.cc:1434] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.039s user 0.005s sys 0.001s
I20260504 13:56:11.997779 9931 tablet_bootstrap.cc:492] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:11.998875 10013 ts_tablet_manager.cc:1434] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.028s user 0.004s sys 0.002s
I20260504 13:56:12.000141 9337 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet b4dca0cadd5e400eab1365c03a8de4f4 (table default.Table_15bf9089e68d4882838fe4f1c65156ef [id=3814e135130a4213a46df3cd93eca49e]). Expected version 1 got 0
I20260504 13:56:12.008734 10013 tablet_bootstrap.cc:492] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:12.011235 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67f26d71286c4f90abd681f9fc9d1e30" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:12.012914 10002 ts_tablet_manager.cc:1929] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:12.013301 10002 log.cc:1199] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/8d25c16864f343a88fa77535831d5a52
I20260504 13:56:11.982179 9944 ts_tablet_manager.cc:1929] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:12.015391 9944 log.cc:1199] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/385cc1f6f4b24c268c25c61a4741480d
I20260504 13:56:12.016088 9944 ts_tablet_manager.cc:1950] T 385cc1f6f4b24c268c25c61a4741480d P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:11.990435 10004 tablet.cc:1719] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6: Skipping requested alter to schema version 2, tablet already version 2
I20260504 13:56:12.018774 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 385cc1f6f4b24c268c25c61a4741480d (table default.Table_659330a3d91d404ebc0d526364ed2f0b [id=0b1606cb64b440cea0b713bd369c440d]) successfully deleted
W20260504 13:56:12.020195 9335 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14b7f1e834d0439c89abb180bd13853d has no consensus state
W20260504 13:56:12.020321 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet c6a6b276cfa14a85a5324728a12b4609 because the tablet was not found. No further retry: Not found: Tablet not found: c6a6b276cfa14a85a5324728a12b4609
I20260504 13:56:12.015012 10002 ts_tablet_manager.cc:1950] T 8d25c16864f343a88fa77535831d5a52 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:12.016005 10049 tablet.cc:1724] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:12.032793 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 8d25c16864f343a88fa77535831d5a52 (table default.Table_c943f49871964cdc9c802e13bfb75928 [id=ed4c94257e234052adc4d311758ebbd3]) successfully deleted
I20260504 13:56:12.036130 9938 raft_consensus.cc:493] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:12.036576 9938 raft_consensus.cc:515] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.038317 9938 leader_election.cc:290] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:12.039533 10002 tablet_replica.cc:333] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:12.039789 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44017e2079be49258bab49325e3da1c1" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:12.040380 10002 raft_consensus.cc:2243] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:12.019996 10013 tablet_bootstrap.cc:654] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:12.041018 9768 leader_election.cc:343] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:12.041342 9932 tablet_bootstrap.cc:492] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:12.040874 10002 raft_consensus.cc:2272] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:12.041797 9932 ts_tablet_manager.cc:1403] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.074s user 0.009s sys 0.001s
I20260504 13:56:12.044243 9932 raft_consensus.cc:359] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.044909 10002 ts_tablet_manager.cc:1916] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:12.046111 9931 tablet_bootstrap.cc:654] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:12.050558 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44017e2079be49258bab49325e3da1c1" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:12.052217 9767 leader_election.cc:343] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:12.054140 9932 raft_consensus.cc:385] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:12.054476 9932 raft_consensus.cc:740] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:12.055219 9932 consensus_queue.cc:260] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.059278 9767 leader_election.cc:304] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:12.060669 10053 tablet.cc:1724] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:12.061368 9938 raft_consensus.cc:2749] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:12.066448 10041 tablet.cc:1724] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:12.067041 10004 tablet.cc:1724] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:12.075809 9932 ts_tablet_manager.cc:1434] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20260504 13:56:12.078841 9932 tablet_bootstrap.cc:492] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:12.079483 10032 raft_consensus.cc:493] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:12.079888 10032 raft_consensus.cc:515] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.081616 10032 leader_election.cc:290] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:12.083009 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f31ed9ad6d147d486c44606e68de22e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
W20260504 13:56:12.084264 9768 leader_election.cc:343] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:12.104177 9932 tablet_bootstrap.cc:654] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:12.088784 10013 tablet_bootstrap.cc:492] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:12.105135 10013 ts_tablet_manager.cc:1403] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.097s user 0.010s sys 0.000s
W20260504 13:56:12.103066 9767 leader_election.cc:343] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:12.105574 9767 leader_election.cc:304] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:12.108294 9938 raft_consensus.cc:2749] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:12.107935 10013 raft_consensus.cc:359] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.108637 10013 raft_consensus.cc:385] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:12.108934 10013 raft_consensus.cc:740] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:12.109695 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 385cc1f6f4b24c268c25c61a4741480d (table default.Table_659330a3d91d404ebc0d526364ed2f0b [id=0b1606cb64b440cea0b713bd369c440d]) successfully deleted
I20260504 13:56:12.111346 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f31ed9ad6d147d486c44606e68de22e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:12.118034 10049 tablet.cc:1719] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00: Skipping requested alter to schema version 2, tablet already version 2
I20260504 13:56:12.118548 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_775e943af0a24713b0aeb8ba35a717c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:12.120126 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_775e943af0a24713b0aeb8ba35a717c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:12.122206 10026 raft_consensus.cc:493] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:12.122654 10026 raft_consensus.cc:515] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.123505 10013 consensus_queue.cc:260] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.124559 10026 leader_election.cc:290] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:12.126837 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4dca0cadd5e400eab1365c03a8de4f4" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:12.127506 9833 raft_consensus.cc:2468] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:12.128140 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4dca0cadd5e400eab1365c03a8de4f4" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:12.129246 9620 leader_election.cc:304] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:12.130055 10026 raft_consensus.cc:2804] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:12.130370 10026 raft_consensus.cc:493] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:12.130684 10026 raft_consensus.cc:3060] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:12.136765 9540 raft_consensus.cc:2468] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:12.137923 10013 ts_tablet_manager.cc:1434] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20260504 13:56:12.138605 9968 consensus_queue.cc:1048] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:12.144443 9932 tablet_bootstrap.cc:492] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:12.144865 9932 ts_tablet_manager.cc:1403] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.000s
I20260504 13:56:12.146944 10026 raft_consensus.cc:515] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.147873 9338 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 987f77386b504f17ab62c796f78f4cb2
W20260504 13:56:12.149602 9767 leader_election.cc:343] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:12.150012 9767 leader_election.cc:304] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:12.152434 9931 tablet_bootstrap.cc:492] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:12.152824 9931 ts_tablet_manager.cc:1403] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.155s user 0.005s sys 0.003s
I20260504 13:56:12.153106 10013 tablet_bootstrap.cc:492] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:12.156075 9515 tablet_service.cc:1558] Processing DeleteTablet for tablet 987f77386b504f17ab62c796f78f4cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:12.100368 10002 ts_tablet_manager.cc:1929] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:12.157497 9938 raft_consensus.cc:2749] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:12.157802 10002 log.cc:1199] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/ddd9d68a1e9d45c9b6f9c8a90912ef54
I20260504 13:56:12.158529 10002 ts_tablet_manager.cc:1950] T ddd9d68a1e9d45c9b6f9c8a90912ef54 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:12.160811 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4dca0cadd5e400eab1365c03a8de4f4" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:12.161299 9540 raft_consensus.cc:3060] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:12.162622 9338 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_4cf6527a29704312924d2bf06660d6d9" } new_table_name: "default.Table_b3c5737b08d240ba961aa3228f09861f" modify_external_catalogs: true
I20260504 13:56:12.147552 9932 raft_consensus.cc:359] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.157526 9662 tablet_service.cc:1558] Processing DeleteTablet for tablet 987f77386b504f17ab62c796f78f4cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:12.163337 9932 raft_consensus.cc:385] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:12.163614 9932 raft_consensus.cc:740] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:12.164356 9932 consensus_queue.cc:260] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:12.164800 9329 catalog_manager.cc:4982] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 987f77386b504f17ab62c796f78f4cb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 987f77386b504f17ab62c796f78f4cb2 already in progress: creating tablet
I20260504 13:56:12.170687 9540 raft_consensus.cc:2468] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:12.173053 9618 leader_election.cc:304] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:12.178031 9811 tablet_service.cc:1558] Processing DeleteTablet for tablet 987f77386b504f17ab62c796f78f4cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:12.180095 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4dca0cadd5e400eab1365c03a8de4f4" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:12.180661 9833 raft_consensus.cc:3060] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:12.187120 9833 raft_consensus.cc:2468] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:12.190883 10026 leader_election.cc:290] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:12.191224 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_7e17be2257534659b742a47c80f3afa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:12.193701 10057 raft_consensus.cc:2804] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:12.194149 10057 raft_consensus.cc:697] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:12.197114 9932 ts_tablet_manager.cc:1434] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.052s user 0.009s sys 0.000s
I20260504 13:56:12.192914 9931 raft_consensus.cc:359] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:12.196707 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e17be2257534659b742a47c80f3afa6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:12.197585 9931 raft_consensus.cc:385] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:12.197868 9931 raft_consensus.cc:740] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:12.199011 9932 tablet_bootstrap.cc:492] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:12.200042 9338 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b7a71dbcc694e1a97019188cfb2c672 consensus state has no leader
I20260504 13:56:12.200440 9931 consensus_queue.cc:260] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:12.203581 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 987f77386b504f17ab62c796f78f4cb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 987f77386b504f17ab62c796f78f4cb2 already in progress: creating tablet
I20260504 13:56:12.203852 10058 raft_consensus.cc:493] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:12.204288 10058 raft_consensus.cc:515] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.211621 10058 leader_election.cc:290] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:12.212698 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb7f58645c76441589368072274ebc8f" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:12.213299 9833 raft_consensus.cc:2468] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:12.214946 10013 tablet_bootstrap.cc:654] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:12.215255 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb7f58645c76441589368072274ebc8f" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:12.215485 9620 leader_election.cc:304] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:12.215838 9540 raft_consensus.cc:2468] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:12.216773 10058 raft_consensus.cc:2804] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:12.217358 10058 raft_consensus.cc:493] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:12.217998 10058 raft_consensus.cc:3060] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:12.214833 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet ddd9d68a1e9d45c9b6f9c8a90912ef54 (table default.Table_eb48d03e69cd40e69ca8ae3e9f3e9dcf [id=0d9014419f714d2c82f3a861926fa0c8]) successfully deleted
I20260504 13:56:12.195073 10057 consensus_queue.cc:237] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:12.237077 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 987f77386b504f17ab62c796f78f4cb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 987f77386b504f17ab62c796f78f4cb2 already in progress: creating tablet
I20260504 13:56:12.237743 10013 tablet_bootstrap.cc:492] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:12.238152 10013 ts_tablet_manager.cc:1403] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.085s user 0.009s sys 0.007s
I20260504 13:56:12.237998 9338 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_e282b7831dc24295974e99bb900ca585" } new_table_name: "default.Table_7111b51688cb4b7c8cc36ff801f7dbcd" modify_external_catalogs: true
I20260504 13:56:12.240502 10013 raft_consensus.cc:359] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.241094 10013 raft_consensus.cc:385] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:12.243237 10013 raft_consensus.cc:740] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:12.245287 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_7be741ea2c0d40d9b2a22d22f349739e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:12.246618 10013 consensus_queue.cc:260] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:12.247025 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7be741ea2c0d40d9b2a22d22f349739e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:12.249405 10044 consensus_queue.cc:1048] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:12.205854 9931 ts_tablet_manager.cc:1434] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.053s user 0.005s sys 0.001s
I20260504 13:56:12.258500 9931 tablet_bootstrap.cc:492] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:12.266175 9335 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet b4dca0cadd5e400eab1365c03a8de4f4 (table default.Table_15bf9089e68d4882838fe4f1c65156ef [id=3814e135130a4213a46df3cd93eca49e]). Expected version 1 got 0
I20260504 13:56:12.266791 9335 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 (table default.Table_0eae04160cf241df95cc53da6f4dfacf [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 2 got 0
I20260504 13:56:12.268266 9340 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: default.Table_0eae04160cf241df95cc53da6f4dfacf [id=3822050e2a9d4a9db8b0e36430fd0b7f] alter complete (version 2)
I20260504 13:56:12.268471 9931 tablet_bootstrap.cc:654] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:12.210361 9932 tablet_bootstrap.cc:654] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:12.273828 9663 tablet_service.cc:1558] Processing DeleteTablet for tablet 8d25c16864f343a88fa77535831d5a52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:10 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:12.273944 9662 tablet_service.cc:1558] Processing DeleteTablet for tablet ddd9d68a1e9d45c9b6f9c8a90912ef54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:12.274997 9661 tablet_service.cc:1511] Processing CreateTablet for tablet e0e3b36e1f654a3a8df51480801dabd7 (DEFAULT_TABLE table=default.Table_e282b7831dc24295974e99bb900ca585 [id=8dcabd7d67334598926f927a62e630d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.271728 10058 raft_consensus.cc:515] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.275596 9520 tablet_service.cc:1511] Processing CreateTablet for tablet e0e3b36e1f654a3a8df51480801dabd7 (DEFAULT_TABLE table=default.Table_e282b7831dc24295974e99bb900ca585 [id=8dcabd7d67334598926f927a62e630d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.276023 9661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e3b36e1f654a3a8df51480801dabd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.276499 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e3b36e1f654a3a8df51480801dabd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.278345 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb7f58645c76441589368072274ebc8f" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:12.278915 9833 raft_consensus.cc:3060] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:12.279459 9944 tablet_replica.cc:333] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:12.280236 9944 raft_consensus.cc:2243] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:12.280200 9811 tablet_service.cc:1511] Processing CreateTablet for tablet e0e3b36e1f654a3a8df51480801dabd7 (DEFAULT_TABLE table=default.Table_e282b7831dc24295974e99bb900ca585 [id=8dcabd7d67334598926f927a62e630d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.280687 9944 raft_consensus.cc:2272] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:12.281201 9811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e3b36e1f654a3a8df51480801dabd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.281852 9662 tablet_service.cc:1511] Processing CreateTablet for tablet d7e556f81ec940f1aa3c55b3a25f31ab (DEFAULT_TABLE table=default.Table_ea18464634544273a0bbb10fce8b88ba [id=2388ca57302443e48f66ac3a8db39000]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.282353 9515 tablet_service.cc:1511] Processing CreateTablet for tablet d7e556f81ec940f1aa3c55b3a25f31ab (DEFAULT_TABLE table=default.Table_ea18464634544273a0bbb10fce8b88ba [id=2388ca57302443e48f66ac3a8db39000]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.282532 9944 ts_tablet_manager.cc:1916] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:12.282855 9662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e556f81ec940f1aa3c55b3a25f31ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.283193 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb7f58645c76441589368072274ebc8f" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:12.283754 9540 raft_consensus.cc:3060] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:12.283380 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e556f81ec940f1aa3c55b3a25f31ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.288829 9814 tablet_service.cc:1511] Processing CreateTablet for tablet d7e556f81ec940f1aa3c55b3a25f31ab (DEFAULT_TABLE table=default.Table_ea18464634544273a0bbb10fce8b88ba [id=2388ca57302443e48f66ac3a8db39000]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.289700 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e556f81ec940f1aa3c55b3a25f31ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.299307 9663 tablet_service.cc:1511] Processing CreateTablet for tablet 0867df9ea115418193b6d9d3fc251923 (DEFAULT_TABLE table=default.Table_39d77929b2fb4ca585977d4c8a362129 [id=92ea89acf1fc427690ee0a61cbbb1647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.300486 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0867df9ea115418193b6d9d3fc251923. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.301721 9514 tablet_service.cc:1511] Processing CreateTablet for tablet 0867df9ea115418193b6d9d3fc251923 (DEFAULT_TABLE table=default.Table_39d77929b2fb4ca585977d4c8a362129 [id=92ea89acf1fc427690ee0a61cbbb1647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.302731 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0867df9ea115418193b6d9d3fc251923. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.306036 9813 tablet_service.cc:1511] Processing CreateTablet for tablet 0867df9ea115418193b6d9d3fc251923 (DEFAULT_TABLE table=default.Table_39d77929b2fb4ca585977d4c8a362129 [id=92ea89acf1fc427690ee0a61cbbb1647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.306981 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0867df9ea115418193b6d9d3fc251923. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.306842 9540 raft_consensus.cc:2468] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:12.308096 9833 raft_consensus.cc:2468] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:12.309074 9620 leader_election.cc:304] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:12.309844 10026 raft_consensus.cc:2804] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:12.310259 10026 raft_consensus.cc:697] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:12.310999 10026 consensus_queue.cc:237] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.318940 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_db02d3a0ad5647ffa14d248cbd3bc76d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:12.320662 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db02d3a0ad5647ffa14d248cbd3bc76d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:12.322988 10058 leader_election.cc:290] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:12.324182 9665 tablet_service.cc:1511] Processing CreateTablet for tablet 4fdc09a498be470ab04563c62caf7755 (DEFAULT_TABLE table=default.Table_49715b69b840473b9f7b0fdabcc0084e [id=700b914cd7ec44e791cd0a27d4b9bf81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.325392 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fdc09a498be470ab04563c62caf7755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.339957 9931 tablet_bootstrap.cc:492] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:12.340343 9931 ts_tablet_manager.cc:1403] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.000s
I20260504 13:56:12.346133 9513 tablet_service.cc:1511] Processing CreateTablet for tablet 4fdc09a498be470ab04563c62caf7755 (DEFAULT_TABLE table=default.Table_49715b69b840473b9f7b0fdabcc0084e [id=700b914cd7ec44e791cd0a27d4b9bf81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.347265 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fdc09a498be470ab04563c62caf7755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.346917 9814 tablet_service.cc:1511] Processing CreateTablet for tablet 4fdc09a498be470ab04563c62caf7755 (DEFAULT_TABLE table=default.Table_49715b69b840473b9f7b0fdabcc0084e [id=700b914cd7ec44e791cd0a27d4b9bf81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.348875 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fdc09a498be470ab04563c62caf7755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.342737 9931 raft_consensus.cc:359] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.352054 9931 raft_consensus.cc:385] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:12.352372 9931 raft_consensus.cc:740] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:12.352833 9666 tablet_service.cc:1511] Processing CreateTablet for tablet bdf1c3cef3334a849f7b5e429b7f4990 (DEFAULT_TABLE table=default.Table_c15418510612433e86cd370f162b6c31 [id=2c6aa17153154189a33f5f3aa00b62e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.353884 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdf1c3cef3334a849f7b5e429b7f4990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.354597 9340 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_ea18464634544273a0bbb10fce8b88ba" } modify_external_catalogs: true
I20260504 13:56:12.355103 9340 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_ea18464634544273a0bbb10fce8b88ba" } modify_external_catalogs: true
I20260504 13:56:12.356590 9931 consensus_queue.cc:260] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.363960 9516 tablet_service.cc:1511] Processing CreateTablet for tablet bdf1c3cef3334a849f7b5e429b7f4990 (DEFAULT_TABLE table=default.Table_c15418510612433e86cd370f162b6c31 [id=2c6aa17153154189a33f5f3aa00b62e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.365012 9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdf1c3cef3334a849f7b5e429b7f4990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.367708 9811 tablet_service.cc:1511] Processing CreateTablet for tablet bdf1c3cef3334a849f7b5e429b7f4990 (DEFAULT_TABLE table=default.Table_c15418510612433e86cd370f162b6c31 [id=2c6aa17153154189a33f5f3aa00b62e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.368847 9811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdf1c3cef3334a849f7b5e429b7f4990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.369808 9931 ts_tablet_manager.cc:1434] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.029s user 0.004s sys 0.003s
I20260504 13:56:12.371865 9665 tablet_service.cc:1511] Processing CreateTablet for tablet 14b7f1e834d0439c89abb180bd13853d (DEFAULT_TABLE table=default.Table_a0a1002687744b879e47ae4fd701d903 [id=2d45d703594b416893908650027e94b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.372898 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14b7f1e834d0439c89abb180bd13853d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.373744 9931 tablet_bootstrap.cc:492] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:12.381928 9517 tablet_service.cc:1511] Processing CreateTablet for tablet 14b7f1e834d0439c89abb180bd13853d (DEFAULT_TABLE table=default.Table_a0a1002687744b879e47ae4fd701d903 [id=2d45d703594b416893908650027e94b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.381999 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_c85fc71384a74a73bfe506eab2b8d9b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:12.382946 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14b7f1e834d0439c89abb180bd13853d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:12.383625 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c85fc71384a74a73bfe506eab2b8d9b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:12.407217 9944 ts_tablet_manager.cc:1929] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:12.411276 10013 ts_tablet_manager.cc:1434] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.173s user 0.001s sys 0.006s
I20260504 13:56:12.412237 10013 tablet_bootstrap.cc:492] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:12.417410 9813 tablet_service.cc:1511] Processing CreateTablet for tablet 14b7f1e834d0439c89abb180bd13853d (DEFAULT_TABLE table=default.Table_a0a1002687744b879e47ae4fd701d903 [id=2d45d703594b416893908650027e94b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.418447 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14b7f1e834d0439c89abb180bd13853d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.411348 9944 log.cc:1199] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/8d25c16864f343a88fa77535831d5a52
I20260504 13:56:12.433244 9944 ts_tablet_manager.cc:1950] T 8d25c16864f343a88fa77535831d5a52 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:12.435874 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 8d25c16864f343a88fa77535831d5a52 (table default.Table_c943f49871964cdc9c802e13bfb75928 [id=ed4c94257e234052adc4d311758ebbd3]) successfully deleted
W20260504 13:56:12.441018 9338 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0e3b36e1f654a3a8df51480801dabd7 consensus state has no leader
W20260504 13:56:12.445245 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet ddd9d68a1e9d45c9b6f9c8a90912ef54 because the tablet was not found. No further retry: Not found: Tablet not found: ddd9d68a1e9d45c9b6f9c8a90912ef54
I20260504 13:56:12.447042 9932 tablet_bootstrap.cc:492] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:12.447463 9932 ts_tablet_manager.cc:1403] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.249s user 0.016s sys 0.003s
I20260504 13:56:12.448439 9336 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_b3c5737b08d240ba961aa3228f09861f" } new_table_name: "default.Table_6aecd26cb4c04baea5252f5dfbb75663" modify_external_catalogs: true
I20260504 13:56:12.450066 9932 raft_consensus.cc:359] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.450665 9932 raft_consensus.cc:385] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:12.450908 9932 raft_consensus.cc:740] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:12.455096 9931 tablet_bootstrap.cc:654] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:12.463413 9932 consensus_queue.cc:260] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.472066 9684 raft_consensus.cc:3060] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:12.470906 9540 raft_consensus.cc:1275] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Refusing update from remote peer d4051d2bf0bf410eb8acd107560df888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 13:56:12.475704 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_4b58a5b3e40c435b91e8fb70062833c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:12.476526 10032 consensus_queue.cc:1048] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:12.477301 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b58a5b3e40c435b91e8fb70062833c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:12.486223 9684 raft_consensus.cc:1275] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Refusing update from remote peer d4051d2bf0bf410eb8acd107560df888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 13:56:12.488375 10032 consensus_queue.cc:1048] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:12.497570 9932 ts_tablet_manager.cc:1434] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.050s user 0.005s sys 0.000s
I20260504 13:56:12.505177 9932 tablet_bootstrap.cc:492] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:12.506953 9340 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet d7e556f81ec940f1aa3c55b3a25f31ab
I20260504 13:56:12.508975 9662 tablet_service.cc:1558] Processing DeleteTablet for tablet d7e556f81ec940f1aa3c55b3a25f31ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:12.509306 9813 tablet_service.cc:1558] Processing DeleteTablet for tablet d7e556f81ec940f1aa3c55b3a25f31ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:56380
W20260504 13:56:12.510671 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet d7e556f81ec940f1aa3c55b3a25f31ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7e556f81ec940f1aa3c55b3a25f31ab already in progress: creating tablet
I20260504 13:56:12.515120 9514 tablet_service.cc:1558] Processing DeleteTablet for tablet d7e556f81ec940f1aa3c55b3a25f31ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:52642
W20260504 13:56:12.517282 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet d7e556f81ec940f1aa3c55b3a25f31ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7e556f81ec940f1aa3c55b3a25f31ab already in progress: creating tablet
W20260504 13:56:12.522382 9329 catalog_manager.cc:4982] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet d7e556f81ec940f1aa3c55b3a25f31ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7e556f81ec940f1aa3c55b3a25f31ab already in progress: creating tablet
W20260504 13:56:12.533964 9336 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b7a71dbcc694e1a97019188cfb2c672 consensus state has no leader
I20260504 13:56:12.470935 10013 tablet_bootstrap.cc:654] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:12.554579 10041 tablet.cc:1724] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:12.566699 9932 tablet_bootstrap.cc:654] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:12.567862 10069 log.cc:927] Time spent T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260504 13:56:12.574352 10004 tablet.cc:1724] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:12.579183 10053 tablet.cc:1724] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:12.592176 9931 tablet_bootstrap.cc:492] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:12.592587 9931 ts_tablet_manager.cc:1403] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.219s user 0.007s sys 0.004s
I20260504 13:56:12.594816 9931 raft_consensus.cc:359] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.599433 9931 raft_consensus.cc:385] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:12.599709 9931 raft_consensus.cc:740] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:12.600299 9931 consensus_queue.cc:260] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.612171 9932 tablet_bootstrap.cc:492] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:12.612565 9932 ts_tablet_manager.cc:1403] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.114s user 0.006s sys 0.006s
I20260504 13:56:12.613910 9931 ts_tablet_manager.cc:1434] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260504 13:56:12.614813 9931 tablet_bootstrap.cc:492] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:12.620798 9932 raft_consensus.cc:359] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.621381 9932 raft_consensus.cc:385] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:12.621644 9932 raft_consensus.cc:740] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:12.623864 10013 tablet_bootstrap.cc:492] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:12.624276 10013 ts_tablet_manager.cc:1403] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.212s user 0.011s sys 0.006s
I20260504 13:56:12.626853 10013 raft_consensus.cc:359] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.627581 10013 raft_consensus.cc:385] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:12.627847 10013 raft_consensus.cc:740] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:12.632727 9932 consensus_queue.cc:260] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.640333 9931 tablet_bootstrap.cc:654] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:12.641412 9337 catalog_manager.cc:5671] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:12.642640 9514 tablet_service.cc:1511] Processing CreateTablet for tablet b37a738f2902402ca76e4edebcdba474 (DEFAULT_TABLE table=default.Table_7e17be2257534659b742a47c80f3afa6 [id=da5c427e75e74ea1ae3a5961ed27e0de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.642777 10013 consensus_queue.cc:260] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.643273 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 39f080a0296b44988e8e51e68453a551 (table default.Table_00c39fd2584d48fb965f7e81908e2ae7 [id=532ece9a0c134c1f8c1c3793638f34c5]). Expected version 1 got 0
I20260504 13:56:12.643549 10044 raft_consensus.cc:493] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:12.643710 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b37a738f2902402ca76e4edebcdba474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.643932 10044 raft_consensus.cc:515] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.643903 9337 catalog_manager.cc:5671] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:12.645042 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet b4dca0cadd5e400eab1365c03a8de4f4 (table default.Table_15bf9089e68d4882838fe4f1c65156ef [id=3814e135130a4213a46df3cd93eca49e]). Expected version 1 got 0
I20260504 13:56:12.647673 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "987f77386b504f17ab62c796f78f4cb2" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
W20260504 13:56:12.649029 9473 leader_election.cc:343] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:12.650695 10044 leader_election.cc:290] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:12.650938 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "987f77386b504f17ab62c796f78f4cb2" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:12.651546 9834 raft_consensus.cc:2468] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:12.653102 9474 leader_election.cc:304] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:12.661194 10044 raft_consensus.cc:2804] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:12.661530 10044 raft_consensus.cc:493] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:12.661862 10044 raft_consensus.cc:3060] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:12.667475 9339 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_00c39fd2584d48fb965f7e81908e2ae7" } new_table_name: "default.Table_d01791667b8a4b77855b70cbed601edb" modify_external_catalogs: true
I20260504 13:56:12.668767 9335 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: default.Table_56891f8543a84e859c39b381625a9daa [id=7b3ef250728944d9998ef2df032f8819] alter complete (version 1)
I20260504 13:56:12.670682 9344 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 6f31ed9ad6d147d486c44606e68de22e
I20260504 13:56:12.672233 9665 tablet_service.cc:1511] Processing CreateTablet for tablet b37a738f2902402ca76e4edebcdba474 (DEFAULT_TABLE table=default.Table_7e17be2257534659b742a47c80f3afa6 [id=da5c427e75e74ea1ae3a5961ed27e0de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.673355 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b37a738f2902402ca76e4edebcdba474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.673619 9516 tablet_service.cc:1558] Processing DeleteTablet for tablet 6f31ed9ad6d147d486c44606e68de22e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:12.674307 10002 tablet_replica.cc:333] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:12.675050 10002 raft_consensus.cc:2243] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:12.675520 10002 raft_consensus.cc:2272] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:12.676349 9814 tablet_service.cc:1558] Processing DeleteTablet for tablet 6f31ed9ad6d147d486c44606e68de22e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:12.677040 10003 tablet_replica.cc:333] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:12.677843 10003 raft_consensus.cc:2243] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:12.678323 10003 raft_consensus.cc:2272] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:12.683367 9662 tablet_service.cc:1558] Processing DeleteTablet for tablet 6f31ed9ad6d147d486c44606e68de22e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:12.683612 9344 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_21845a95c3ab4388b869666380989ec3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:12.684697 9932 ts_tablet_manager.cc:1434] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.072s user 0.005s sys 0.001s
W20260504 13:56:12.684875 9329 catalog_manager.cc:4982] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 6f31ed9ad6d147d486c44606e68de22e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f31ed9ad6d147d486c44606e68de22e already in progress: creating tablet
W20260504 13:56:12.685024 9344 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21845a95c3ab4388b869666380989ec3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:12.687932 10002 ts_tablet_manager.cc:1916] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:12.689705 10003 ts_tablet_manager.cc:1916] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:12.702303 9932 tablet_bootstrap.cc:492] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:12.682811 10044 raft_consensus.cc:515] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.706701 9662 tablet_service.cc:1511] Processing CreateTablet for tablet b3793fe8f1ec43ad809403c4593f5ef6 (DEFAULT_TABLE table=default.Table_61add6c7b0dd46b08c0e39ad6d697e8c [id=1749c1a022544b1185fb3d4a4a530afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.707814 9662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3793fe8f1ec43ad809403c4593f5ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.709929 10013 ts_tablet_manager.cc:1434] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.085s user 0.006s sys 0.001s
I20260504 13:56:12.710822 10013 tablet_bootstrap.cc:492] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:12.711586 10044 leader_election.cc:290] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:12.712460 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "987f77386b504f17ab62c796f78f4cb2" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
W20260504 13:56:12.713565 9473 leader_election.cc:343] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:12.715512 9339 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_61add6c7b0dd46b08c0e39ad6d697e8c" } modify_external_catalogs: true
I20260504 13:56:12.716044 9339 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_61add6c7b0dd46b08c0e39ad6d697e8c" } modify_external_catalogs: true
I20260504 13:56:12.716676 10004 tablet.cc:1724] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:12.719270 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_35722081733246ae8ca41a5303921ffe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:12.720937 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35722081733246ae8ca41a5303921ffe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:12.721922 9516 tablet_service.cc:1511] Processing CreateTablet for tablet b3793fe8f1ec43ad809403c4593f5ef6 (DEFAULT_TABLE table=default.Table_61add6c7b0dd46b08c0e39ad6d697e8c [id=1749c1a022544b1185fb3d4a4a530afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.722990 9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3793fe8f1ec43ad809403c4593f5ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.739099 9336 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_7be741ea2c0d40d9b2a22d22f349739e" } new_table_name: "default.Table_3bed4e09fe9d4eceb3bc1614c4cc26bf" modify_external_catalogs: true
I20260504 13:56:12.748044 9342 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 39f080a0296b44988e8e51e68453a551 (table default.Table_d01791667b8a4b77855b70cbed601edb [id=532ece9a0c134c1f8c1c3793638f34c5]). Expected version 2 got 1
I20260504 13:56:12.752847 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_038807318bcc4110b77de9582f6c42f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:12.754539 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_038807318bcc4110b77de9582f6c42f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:12.755220 10053 tablet.cc:1724] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:12.769130 9342 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 39f080a0296b44988e8e51e68453a551 (table default.Table_d01791667b8a4b77855b70cbed601edb [id=532ece9a0c134c1f8c1c3793638f34c5]). Expected version 2 got 1
I20260504 13:56:12.770037 9335 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_ae6e5a6234d14d9a9396afc9966be063"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:12.772961 9335 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae6e5a6234d14d9a9396afc9966be063 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:12.774158 9661 tablet_service.cc:1558] Processing DeleteTablet for tablet 6f31ed9ad6d147d486c44606e68de22e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:12.774884 9944 tablet_replica.cc:333] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:12.775604 9944 raft_consensus.cc:2243] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:12.776015 9944 raft_consensus.cc:2272] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:12.776137 9341 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 39f080a0296b44988e8e51e68453a551 (table default.Table_d01791667b8a4b77855b70cbed601edb [id=532ece9a0c134c1f8c1c3793638f34c5]). Expected version 2 got 1
I20260504 13:56:12.777976 9944 ts_tablet_manager.cc:1916] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:12.713991 10041 tablet.cc:1724] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:12.717656 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "987f77386b504f17ab62c796f78f4cb2" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:12.784338 9834 raft_consensus.cc:3060] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:12.785652 9513 tablet_service.cc:1511] Processing CreateTablet for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (DEFAULT_TABLE table=default.Table_b97e5fa65d9e40c9982c796a6ca8afe1 [id=c058bce8e15341d0bcbdd6a6b537855c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.786412 9339 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet b3793fe8f1ec43ad809403c4593f5ef6
I20260504 13:56:12.786391 9661 tablet_service.cc:1511] Processing CreateTablet for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (DEFAULT_TABLE table=default.Table_b97e5fa65d9e40c9982c796a6ca8afe1 [id=c058bce8e15341d0bcbdd6a6b537855c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.787897 9517 tablet_service.cc:1558] Processing DeleteTablet for tablet b3793fe8f1ec43ad809403c4593f5ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:12.788303 9666 tablet_service.cc:1558] Processing DeleteTablet for tablet b3793fe8f1ec43ad809403c4593f5ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:12.788991 9661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed4f5ea230a413a8dcdbb661ce173b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.792847 9932 tablet_bootstrap.cc:654] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:12.719666 9814 tablet_service.cc:1511] Processing CreateTablet for tablet b37a738f2902402ca76e4edebcdba474 (DEFAULT_TABLE table=default.Table_7e17be2257534659b742a47c80f3afa6 [id=da5c427e75e74ea1ae3a5961ed27e0de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.793798 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b37a738f2902402ca76e4edebcdba474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.759976 9811 tablet_service.cc:1558] Processing DeleteTablet for tablet 987f77386b504f17ab62c796f78f4cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:12.782976 9813 tablet_service.cc:1511] Processing CreateTablet for tablet b3793fe8f1ec43ad809403c4593f5ef6 (DEFAULT_TABLE table=default.Table_61add6c7b0dd46b08c0e39ad6d697e8c [id=1749c1a022544b1185fb3d4a4a530afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.796831 9931 tablet_bootstrap.cc:492] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:12.797212 9931 ts_tablet_manager.cc:1403] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.183s user 0.008s sys 0.001s
I20260504 13:56:12.797490 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3793fe8f1ec43ad809403c4593f5ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.801103 9931 raft_consensus.cc:359] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.801735 9931 raft_consensus.cc:385] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:12.801990 9931 raft_consensus.cc:740] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:12.802750 9931 consensus_queue.cc:260] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.808068 9517 tablet_service.cc:1511] Processing CreateTablet for tablet de639fda874248f0853e74ee60aaf758 (DEFAULT_TABLE table=default.Table_fa64d5b0eb084f2a9ab924aa2cdf3a1c [id=666099fdb1a94091ac6fd99cf58f18e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.810393 9931 ts_tablet_manager.cc:1434] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260504 13:56:12.811102 9931 tablet_bootstrap.cc:492] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:12.812839 9665 tablet_service.cc:1511] Processing CreateTablet for tablet de639fda874248f0853e74ee60aaf758 (DEFAULT_TABLE table=default.Table_fa64d5b0eb084f2a9ab924aa2cdf3a1c [id=666099fdb1a94091ac6fd99cf58f18e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.813840 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de639fda874248f0853e74ee60aaf758. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:12.814496 9336 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet afee197774db411d97468913a207703a has no consensus state
I20260504 13:56:12.816733 9812 tablet_service.cc:1558] Processing DeleteTablet for tablet b3793fe8f1ec43ad809403c4593f5ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:12.817229 9811 tablet_service.cc:1511] Processing CreateTablet for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (DEFAULT_TABLE table=default.Table_b97e5fa65d9e40c9982c796a6ca8afe1 [id=c058bce8e15341d0bcbdd6a6b537855c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.819134 9811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed4f5ea230a413a8dcdbb661ce173b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.821900 10013 tablet_bootstrap.cc:654] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:12.824184 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed4f5ea230a413a8dcdbb661ce173b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.829205 9520 tablet_service.cc:1511] Processing CreateTablet for tablet a43cdb767fd14ea09b4db4d2881b69aa (DEFAULT_TABLE table=default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.830076 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43cdb767fd14ea09b4db4d2881b69aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.834650 9812 tablet_service.cc:1511] Processing CreateTablet for tablet de639fda874248f0853e74ee60aaf758 (DEFAULT_TABLE table=default.Table_fa64d5b0eb084f2a9ab924aa2cdf3a1c [id=666099fdb1a94091ac6fd99cf58f18e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.839421 9812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de639fda874248f0853e74ee60aaf758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.846619 9515 tablet_service.cc:1558] Processing DeleteTablet for tablet 987f77386b504f17ab62c796f78f4cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:12.848614 10057 raft_consensus.cc:493] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:12.849063 10057 raft_consensus.cc:515] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.852056 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67f26d71286c4f90abd681f9fc9d1e30" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:12.853403 9618 leader_election.cc:343] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:12.863575 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de639fda874248f0853e74ee60aaf758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.876346 10004 tablet.cc:1719] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: Skipping requested alter to schema version 2, tablet already version 2
I20260504 13:56:12.876765 9340 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_db02d3a0ad5647ffa14d248cbd3bc76d" } modify_external_catalogs: true
I20260504 13:56:12.877279 9340 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_db02d3a0ad5647ffa14d248cbd3bc76d" } modify_external_catalogs: true
I20260504 13:56:12.882719 10041 tablet.cc:1719] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888: Skipping requested alter to schema version 2, tablet already version 2
I20260504 13:56:12.886819 10057 leader_election.cc:290] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:12.891368 9932 tablet_bootstrap.cc:492] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:12.891736 9932 ts_tablet_manager.cc:1403] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.190s user 0.009s sys 0.000s
I20260504 13:56:12.894142 10053 tablet.cc:1719] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00: Skipping requested alter to schema version 2, tablet already version 2
I20260504 13:56:12.894044 9932 raft_consensus.cc:359] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.894641 9932 raft_consensus.cc:385] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:12.895146 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67f26d71286c4f90abd681f9fc9d1e30" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:12.895730 9833 raft_consensus.cc:2468] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:12.896885 9620 leader_election.cc:304] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:12.901216 9344 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: default.Table_d01791667b8a4b77855b70cbed601edb [id=532ece9a0c134c1f8c1c3793638f34c5] alter complete (version 2)
I20260504 13:56:12.902882 10057 raft_consensus.cc:2804] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:12.903311 10057 raft_consensus.cc:493] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:12.903681 10057 raft_consensus.cc:3060] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:12.904616 9810 tablet_service.cc:1558] Processing DeleteTablet for tablet 03818e34ac0545be880c873cf82cd4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:12.906291 9344 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_7111b51688cb4b7c8cc36ff801f7dbcd" } new_table_name: "default.Table_9e9ef3353dcf4b1fadaa2f51bc8f7041" modify_external_catalogs: true
I20260504 13:56:12.914165 9932 raft_consensus.cc:740] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:12.914866 9932 consensus_queue.cc:260] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.919806 9834 raft_consensus.cc:2468] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:12.921066 9474 leader_election.cc:304] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:12.921801 10044 raft_consensus.cc:2804] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:12.922197 10044 raft_consensus.cc:697] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:12.922549 9662 tablet_service.cc:1511] Processing CreateTablet for tablet a43cdb767fd14ea09b4db4d2881b69aa (DEFAULT_TABLE table=default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.922945 10044 consensus_queue.cc:237] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.926590 9340 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 24ad2abca0e74321936435f0eee083cf
W20260504 13:56:12.933427 9330 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet afee197774db411d97468913a207703a (table default.Table_3bed4e09fe9d4eceb3bc1614c4cc26bf [id=a2697dba5f38435c81dc52316712917d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet afee197774db411d97468913a207703a has no consensus state
I20260504 13:56:12.934388 10013 tablet_bootstrap.cc:492] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:12.939353 9932 ts_tablet_manager.cc:1434] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20260504 13:56:12.940037 9931 tablet_bootstrap.cc:654] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:12.943503 10013 ts_tablet_manager.cc:1403] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.224s user 0.010s sys 0.000s
I20260504 13:56:12.947748 9341 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 0b7a71dbcc694e1a97019188cfb2c672 (table default.Table_6aecd26cb4c04baea5252f5dfbb75663 [id=0649fc4627314bca8b51d0e1f5022e6d]). Expected version 3 got 0
I20260504 13:56:12.955253 9944 ts_tablet_manager.cc:1929] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:12.955624 9944 log.cc:1199] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/6f31ed9ad6d147d486c44606e68de22e
I20260504 13:56:12.956028 10032 raft_consensus.cc:493] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:12.956282 9944 ts_tablet_manager.cc:1950] T 6f31ed9ad6d147d486c44606e68de22e P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:12.956459 10032 raft_consensus.cc:515] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.959091 10003 ts_tablet_manager.cc:1929] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:12.960289 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b320a37b29844c7593a12000eaa3da43" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:12.961933 10013 raft_consensus.cc:359] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.962548 10013 raft_consensus.cc:385] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:12.962811 10013 raft_consensus.cc:740] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:12.963469 10013 consensus_queue.cc:260] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:12.969172 9517 tablet_service.cc:1511] Processing CreateTablet for tablet 69ed77782cd34274932957106ce26766 (DEFAULT_TABLE table=default.Table_cdaaea60f9354320bc4cfee6729ba12f [id=55db3ce6e7694cca94a1705fb1c0e90e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.969571 9666 tablet_service.cc:1511] Processing CreateTablet for tablet 69ed77782cd34274932957106ce26766 (DEFAULT_TABLE table=default.Table_cdaaea60f9354320bc4cfee6729ba12f [id=55db3ce6e7694cca94a1705fb1c0e90e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.970317 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69ed77782cd34274932957106ce26766. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.959697 10032 leader_election.cc:290] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:12.961632 9767 leader_election.cc:343] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:12.973738 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b320a37b29844c7593a12000eaa3da43" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:12.974416 10013 ts_tablet_manager.cc:1434] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.031s user 0.004s sys 0.000s
I20260504 13:56:12.975741 9813 tablet_service.cc:1511] Processing CreateTablet for tablet 69ed77782cd34274932957106ce26766 (DEFAULT_TABLE table=default.Table_cdaaea60f9354320bc4cfee6729ba12f [id=55db3ce6e7694cca94a1705fb1c0e90e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:12.976616 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69ed77782cd34274932957106ce26766. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:12.976745 9768 leader_election.cc:343] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:12.977267 9768 leader_election.cc:304] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:12.980433 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69ed77782cd34274932957106ce26766. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.980260 9343 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_a4d8b2884f1e402fa29eee35f1c46d0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:12.981057 10032 raft_consensus.cc:2749] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:12.981375 10003 log.cc:1199] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/6f31ed9ad6d147d486c44606e68de22e
W20260504 13:56:12.981848 9343 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4d8b2884f1e402fa29eee35f1c46d0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:12.983804 10013 tablet_bootstrap.cc:492] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:12.986050 10002 ts_tablet_manager.cc:1929] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:12.986433 10002 log.cc:1199] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/6f31ed9ad6d147d486c44606e68de22e
I20260504 13:56:12.987454 10003 ts_tablet_manager.cc:1950] T 6f31ed9ad6d147d486c44606e68de22e P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:12.991967 9335 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_cdaaea60f9354320bc4cfee6729ba12f" } modify_external_catalogs: true
I20260504 13:56:12.992470 9335 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_cdaaea60f9354320bc4cfee6729ba12f" } modify_external_catalogs: true
I20260504 13:56:12.996527 9662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43cdb767fd14ea09b4db4d2881b69aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:12.996863 9932 tablet_bootstrap.cc:492] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:12.993539 9931 tablet_bootstrap.cc:492] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:12.999805 9931 ts_tablet_manager.cc:1403] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.189s user 0.008s sys 0.004s
I20260504 13:56:13.002211 9931 raft_consensus.cc:359] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.002936 9931 raft_consensus.cc:385] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.003259 9931 raft_consensus.cc:740] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:12.995779 9814 tablet_service.cc:1511] Processing CreateTablet for tablet a43cdb767fd14ea09b4db4d2881b69aa (DEFAULT_TABLE table=default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.007046 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 6f31ed9ad6d147d486c44606e68de22e (table default.Table_e864e83fbba04cc3abbf7a6e955ac912 [id=df5f0bd632b2428492a2954070269840]) successfully deleted
I20260504 13:56:13.003836 9931 consensus_queue.cc:260] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.009270 10013 tablet_bootstrap.cc:654] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:13.011045 9344 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0e3b36e1f654a3a8df51480801dabd7 consensus state has no leader
I20260504 13:56:13.013981 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_9b8d10cfdddc499a86ba289a87dbe689"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:13.017355 9336 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_ae6e5a6234d14d9a9396afc9966be063" } new_table_name: "default.Table_2abb0f27da024032b5646af14e0946e8" modify_external_catalogs: true
I20260504 13:56:13.013203 10044 raft_consensus.cc:493] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:13.018887 10044 raft_consensus.cc:515] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.015714 10003 tablet_replica.cc:333] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
W20260504 13:56:13.020195 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b8d10cfdddc499a86ba289a87dbe689 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:13.020752 10003 raft_consensus.cc:2243] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:13.020766 10044 leader_election.cc:290] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:13.021175 10003 raft_consensus.cc:2272] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:13.016844 9931 ts_tablet_manager.cc:1434] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260504 13:56:13.021816 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67f26d71286c4f90abd681f9fc9d1e30" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:13.022421 9834 raft_consensus.cc:2468] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:13.023489 9474 leader_election.cc:304] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:13.018687 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43cdb767fd14ea09b4db4d2881b69aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.024179 10044 raft_consensus.cc:2804] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:13.024484 10044 raft_consensus.cc:493] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:13.024789 10044 raft_consensus.cc:3060] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:13.026429 9931 tablet_bootstrap.cc:492] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:12.987062 10002 ts_tablet_manager.cc:1950] T 6f31ed9ad6d147d486c44606e68de22e P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:13.034714 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 6f31ed9ad6d147d486c44606e68de22e (table default.Table_e864e83fbba04cc3abbf7a6e955ac912 [id=df5f0bd632b2428492a2954070269840]) successfully deleted
I20260504 13:56:13.032112 10044 raft_consensus.cc:515] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.041149 10003 ts_tablet_manager.cc:1916] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:13.048712 10044 leader_election.cc:290] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:13.050078 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67f26d71286c4f90abd681f9fc9d1e30" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:13.050611 9834 raft_consensus.cc:3060] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:13.056437 10002 tablet_replica.cc:333] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:13.056815 9834 raft_consensus.cc:2468] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:13.057224 10002 raft_consensus.cc:2243] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:13.057497 10013 tablet_bootstrap.cc:492] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:13.057893 10013 ts_tablet_manager.cc:1403] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.000s
I20260504 13:56:13.057934 9474 leader_election.cc:304] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:13.058729 9931 tablet_bootstrap.cc:654] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:13.059063 10044 raft_consensus.cc:2804] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:13.059631 10044 raft_consensus.cc:697] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:13.060443 10044 consensus_queue.cc:237] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.065043 9147 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
/tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.253:36337
--webserver_interface=127.8.238.253
--webserver_port=44077
--builtin_ntp_servers=127.8.238.212:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.238.254:36145,127.8.238.253:36337,127.8.238.252:44781
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260504 13:56:13.093585 9341 catalog_manager.cc:5671] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:13.095108 9344 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_276161949f364064b8702a62c6beb9e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:13.095834 9931 tablet_bootstrap.cc:492] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:13.096189 9931 ts_tablet_manager.cc:1403] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.070s user 0.011s sys 0.000s
W20260504 13:56:13.096627 9344 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_276161949f364064b8702a62c6beb9e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:13.102173 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet b3793fe8f1ec43ad809403c4593f5ef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3793fe8f1ec43ad809403c4593f5ef6 already in progress: creating tablet
I20260504 13:56:13.104995 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_963c93d5d8ed49e6a2052fa1e652da91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:13.105300 9931 raft_consensus.cc:359] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.105901 9931 raft_consensus.cc:385] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.106127 9931 raft_consensus.cc:740] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
W20260504 13:56:13.106647 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_963c93d5d8ed49e6a2052fa1e652da91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:13.106659 9931 consensus_queue.cc:260] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.109302 10003 ts_tablet_manager.cc:1929] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:13.117559 9335 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 69ed77782cd34274932957106ce26766
I20260504 13:56:13.121119 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 6f31ed9ad6d147d486c44606e68de22e (table default.Table_e864e83fbba04cc3abbf7a6e955ac912 [id=df5f0bd632b2428492a2954070269840]) successfully deleted
W20260504 13:56:13.121737 9329 catalog_manager.cc:4982] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet b3793fe8f1ec43ad809403c4593f5ef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3793fe8f1ec43ad809403c4593f5ef6 already in progress: creating tablet
I20260504 13:56:13.057885 10002 pending_rounds.cc:70] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00: Trying to abort 1 pending ops.
I20260504 13:56:13.123539 10002 pending_rounds.cc:77] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290577099788288 op_type: NO_OP noop_request { }
I20260504 13:56:13.123901 10002 raft_consensus.cc:2889] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:13.124265 10002 raft_consensus.cc:2272] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:13.125128 9335 master_service.cc:954] ReplaceTablet: received request to replace tablet 0867df9ea115418193b6d9d3fc251923 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:13.120404 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67f26d71286c4f90abd681f9fc9d1e30" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:13.128113 10002 ts_tablet_manager.cc:1916] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:13.132052 9664 tablet_service.cc:1558] Processing DeleteTablet for tablet 69ed77782cd34274932957106ce26766 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:13.133268 9343 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_d4a59933eb124b4295cc0b066a46f9ea" } modify_external_catalogs: true
I20260504 13:56:13.133750 9343 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_d4a59933eb124b4295cc0b066a46f9ea" } modify_external_catalogs: true
I20260504 13:56:13.134739 9540 raft_consensus.cc:1275] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Refusing update from remote peer 5beceb6317444247b31b44a6581707c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 13:56:13.128973 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67f26d71286c4f90abd681f9fc9d1e30" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:13.134964 9661 tablet_service.cc:1558] Processing DeleteTablet for tablet 987f77386b504f17ab62c796f78f4cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:13.143747 10077 consensus_queue.cc:1048] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:13.061767 10013 raft_consensus.cc:359] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.144910 10013 raft_consensus.cc:385] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.145159 10013 raft_consensus.cc:740] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:13.122514 9811 tablet_service.cc:1558] Processing DeleteTablet for tablet 69ed77782cd34274932957106ce26766 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:13.145762 10003 log.cc:1199] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/987f77386b504f17ab62c796f78f4cb2
I20260504 13:56:13.146482 10003 ts_tablet_manager.cc:1950] T 987f77386b504f17ab62c796f78f4cb2 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:13.147051 9339 master_service.cc:954] ReplaceTablet: received request to replace tablet 67f26d71286c4f90abd681f9fc9d1e30 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:13.137557 9931 ts_tablet_manager.cc:1434] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20260504 13:56:13.147059 10057 raft_consensus.cc:515] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.150164 9931 tablet_bootstrap.cc:492] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:13.153170 10013 consensus_queue.cc:260] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.156280 10013 ts_tablet_manager.cc:1434] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.098s user 0.006s sys 0.000s
W20260504 13:56:13.159283 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 69ed77782cd34274932957106ce26766 because the tablet was not found. No further retry: Not found: Tablet not found: 69ed77782cd34274932957106ce26766
I20260504 13:56:13.160650 10013 tablet_bootstrap.cc:492] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:13.163542 10032 raft_consensus.cc:493] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:13.167534 10032 raft_consensus.cc:515] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.157315 10087 raft_consensus.cc:493] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:13.173043 9684 raft_consensus.cc:2393] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 1: Already voted for candidate 5beceb6317444247b31b44a6581707c6 in this term.
I20260504 13:56:13.173187 10087 raft_consensus.cc:515] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.175783 10057 leader_election.cc:290] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:13.157933 9944 tablet_replica.cc:333] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:13.165678 9932 tablet_bootstrap.cc:654] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:13.176635 9944 raft_consensus.cc:2243] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:13.177101 9944 raft_consensus.cc:2272] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:13.175539 10087 leader_election.cc:290] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:13.179219 9944 ts_tablet_manager.cc:1916] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:13.180725 9520 tablet_service.cc:1558] Processing DeleteTablet for tablet 69ed77782cd34274932957106ce26766 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:13.190963 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44017e2079be49258bab49325e3da1c1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:13.191771 9539 raft_consensus.cc:2468] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:13.192898 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67f26d71286c4f90abd681f9fc9d1e30" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:13.195035 9618 leader_election.cc:304] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:13.196395 10087 raft_consensus.cc:2804] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:13.196826 10087 raft_consensus.cc:493] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:13.197702 10087 raft_consensus.cc:3060] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:13.209736 10077 consensus_queue.cc:1048] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:13.211549 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91371bf848f5465ea2db0c090583382d" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:13.212188 9834 raft_consensus.cc:1275] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Refusing update from remote peer 5beceb6317444247b31b44a6581707c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260504 13:56:13.212824 9767 leader_election.cc:343] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:13.213742 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 987f77386b504f17ab62c796f78f4cb2 (table default.Table_a3c5361bede04e74b57b95480a479298 [id=6949858f6acf4cec88b9af9791e5c693]) successfully deleted
I20260504 13:56:13.215232 9938 raft_consensus.cc:493] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:13.215780 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44017e2079be49258bab49325e3da1c1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:13.215746 9338 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_b97e5fa65d9e40c9982c796a6ca8afe1" } new_table_name: "default.Table_505ad6df21dd404d8ac2d5b4c150271d" modify_external_catalogs: true
I20260504 13:56:13.215649 9938 raft_consensus.cc:515] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.216293 9833 raft_consensus.cc:2468] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:13.215091 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67f26d71286c4f90abd681f9fc9d1e30" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:13.217422 9832 raft_consensus.cc:2393] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5beceb6317444247b31b44a6581707c6 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:13.217415 9938 leader_election.cc:290] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:13.217973 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91371bf848f5465ea2db0c090583382d" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:13.219822 9620 leader_election.cc:304] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:13.220607 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03818e34ac0545be880c873cf82cd4ea" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:13.220798 10077 raft_consensus.cc:2749] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260504 13:56:13.222724 10003 tablet_replica.cc:333] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:13.222719 10077 consensus_queue.cc:1048] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:13.223665 10003 raft_consensus.cc:2243] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:13.224156 10003 raft_consensus.cc:2272] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:13.224774 9767 leader_election.cc:343] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:13.225703 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet b3793fe8f1ec43ad809403c4593f5ef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3793fe8f1ec43ad809403c4593f5ef6 already in progress: creating tablet
I20260504 13:56:13.227021 9931 tablet_bootstrap.cc:654] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:13.230187 10002 ts_tablet_manager.cc:1929] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:13.230562 10002 log.cc:1199] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/987f77386b504f17ab62c796f78f4cb2
I20260504 13:56:13.231101 10032 leader_election.cc:290] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:13.231280 10002 ts_tablet_manager.cc:1950] T 987f77386b504f17ab62c796f78f4cb2 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:13.233453 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 987f77386b504f17ab62c796f78f4cb2 (table default.Table_a3c5361bede04e74b57b95480a479298 [id=6949858f6acf4cec88b9af9791e5c693]) successfully deleted
I20260504 13:56:13.234889 9932 tablet_bootstrap.cc:492] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:13.238029 9932 ts_tablet_manager.cc:1403] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.241s user 0.008s sys 0.006s
I20260504 13:56:13.239114 10013 tablet_bootstrap.cc:654] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:13.241305 9932 raft_consensus.cc:359] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:13.248747 9768 leader_election.cc:343] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:13.249297 10003 ts_tablet_manager.cc:1916] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:13.258935 9932 raft_consensus.cc:385] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.259218 9932 raft_consensus.cc:740] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:13.259757 9932 consensus_queue.cc:260] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.261446 9932 ts_tablet_manager.cc:1434] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20260504 13:56:13.245995 9944 ts_tablet_manager.cc:1929] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:13.248140 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03818e34ac0545be880c873cf82cd4ea" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:13.252447 10087 raft_consensus.cc:515] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.264583 10087 leader_election.cc:290] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:13.264976 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44017e2079be49258bab49325e3da1c1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:13.265585 9539 raft_consensus.cc:3060] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:13.265079 10087 consensus_queue.cc:1048] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:13.266112 9944 log.cc:1199] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/987f77386b504f17ab62c796f78f4cb2
I20260504 13:56:13.266752 9944 ts_tablet_manager.cc:1950] T 987f77386b504f17ab62c796f78f4cb2 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
W20260504 13:56:13.268203 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 69ed77782cd34274932957106ce26766 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69ed77782cd34274932957106ce26766 already in progress: creating tablet
I20260504 13:56:13.270417 9932 tablet_bootstrap.cc:492] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:13.274354 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 987f77386b504f17ab62c796f78f4cb2 (table default.Table_a3c5361bede04e74b57b95480a479298 [id=6949858f6acf4cec88b9af9791e5c693]) successfully deleted
I20260504 13:56:13.249199 9768 leader_election.cc:304] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
W20260504 13:56:13.276897 9768 leader_election.cc:343] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:13.277277 9768 leader_election.cc:304] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
W20260504 13:56:13.277825 9768 raft_consensus.cc:2686] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260504 13:56:13.278512 9938 raft_consensus.cc:2749] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:13.278378 10004 tablet.cc:1724] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:13.275738 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44017e2079be49258bab49325e3da1c1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:13.279536 9834 raft_consensus.cc:3060] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:13.284056 9932 tablet_bootstrap.cc:654] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:13.285696 10041 tablet.cc:1724] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:13.292668 10053 tablet.cc:1724] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:13.306329 9938 raft_consensus.cc:493] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:13.306778 9938 raft_consensus.cc:515] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.308537 9938 leader_election.cc:290] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:13.310901 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b7a71dbcc694e1a97019188cfb2c672" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:13.312399 10013 tablet_bootstrap.cc:492] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:13.312752 10013 ts_tablet_manager.cc:1403] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.152s user 0.008s sys 0.000s
W20260504 13:56:13.313063 9768 leader_election.cc:343] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:13.315305 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b7a71dbcc694e1a97019188cfb2c672" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:13.316476 9767 leader_election.cc:343] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:13.316869 9767 leader_election.cc:304] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:13.317919 9938 raft_consensus.cc:2749] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:13.322026 9539 raft_consensus.cc:2468] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:13.329994 9932 tablet_bootstrap.cc:492] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:13.330459 9932 ts_tablet_manager.cc:1403] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.003s
I20260504 13:56:13.332764 9932 raft_consensus.cc:359] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.333371 9932 raft_consensus.cc:385] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.333611 9932 raft_consensus.cc:740] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:13.340276 9931 tablet_bootstrap.cc:492] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:13.340651 9931 ts_tablet_manager.cc:1403] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.193s user 0.009s sys 0.004s
I20260504 13:56:13.342840 9931 raft_consensus.cc:359] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.343451 9931 raft_consensus.cc:385] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.343686 9931 raft_consensus.cc:740] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:13.343650 9618 leader_election.cc:304] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:13.324684 10013 raft_consensus.cc:359] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.344230 10013 raft_consensus.cc:385] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.344367 10057 raft_consensus.cc:2804] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:13.344507 10013 raft_consensus.cc:740] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:13.344805 10057 raft_consensus.cc:697] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:13.345127 10013 consensus_queue.cc:260] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.345520 10057 consensus_queue.cc:237] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.347218 10013 ts_tablet_manager.cc:1434] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.034s user 0.006s sys 0.002s
I20260504 13:56:13.348170 10013 tablet_bootstrap.cc:492] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:13.349983 9932 consensus_queue.cc:260] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.357172 9932 ts_tablet_manager.cc:1434] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.026s user 0.001s sys 0.005s
I20260504 13:56:13.352005 9931 consensus_queue.cc:260] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.372315 9931 ts_tablet_manager.cc:1434] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.031s user 0.005s sys 0.000s
I20260504 13:56:13.373631 9932 tablet_bootstrap.cc:492] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:13.376557 9938 raft_consensus.cc:493] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:13.377005 9938 raft_consensus.cc:515] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.378157 9931 tablet_bootstrap.cc:492] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:13.378861 9938 leader_election.cc:290] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:13.381327 10013 tablet_bootstrap.cc:654] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:13.382522 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867df9ea115418193b6d9d3fc251923" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:13.383548 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867df9ea115418193b6d9d3fc251923" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
W20260504 13:56:13.385047 9768 leader_election.cc:343] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:13.386343 9767 leader_election.cc:343] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:13.386732 9767 leader_election.cc:304] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:13.387779 10091 raft_consensus.cc:2749] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:13.392541 9931 tablet_bootstrap.cc:654] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:13.393801 9932 tablet_bootstrap.cc:654] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:13.411782 9336 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet baf07a970a7c4a35b3c849b06d6c88d0 has no consensus state
I20260504 13:56:13.415563 9932 tablet_bootstrap.cc:492] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:13.415926 9932 ts_tablet_manager.cc:1403] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.003s
I20260504 13:56:13.418274 9932 raft_consensus.cc:359] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.418856 9932 raft_consensus.cc:385] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.422750 9931 tablet_bootstrap.cc:492] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:13.423107 9931 ts_tablet_manager.cc:1403] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.000s
I20260504 13:56:13.425331 9931 raft_consensus.cc:359] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.433362 9931 raft_consensus.cc:385] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.433635 9931 raft_consensus.cc:740] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:13.419083 9932 raft_consensus.cc:740] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:13.435361 9932 consensus_queue.cc:260] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.436578 9931 consensus_queue.cc:260] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.442651 9932 ts_tablet_manager.cc:1434] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
W20260504 13:56:13.445010 10100 log.cc:927] Time spent T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260504 13:56:13.401825 9834 raft_consensus.cc:2468] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:13.453141 9932 tablet_bootstrap.cc:492] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:13.451023 9931 ts_tablet_manager.cc:1434] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.028s user 0.004s sys 0.000s
I20260504 13:56:13.461303 9931 tablet_bootstrap.cc:492] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:13.471127 9932 tablet_bootstrap.cc:654] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:13.473922 10095 raft_consensus.cc:493] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:13.474375 10095 raft_consensus.cc:515] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.475759 10013 tablet_bootstrap.cc:492] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:13.476122 10013 ts_tablet_manager.cc:1403] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.128s user 0.007s sys 0.012s
I20260504 13:56:13.476351 10095 leader_election.cc:290] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:13.477836 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b320a37b29844c7593a12000eaa3da43" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:13.478552 9832 raft_consensus.cc:2468] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:13.478658 10013 raft_consensus.cc:359] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.479278 10013 raft_consensus.cc:385] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.479599 10013 raft_consensus.cc:740] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:13.479792 9620 leader_election.cc:304] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:13.480551 10095 raft_consensus.cc:2804] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:13.480304 10013 consensus_queue.cc:260] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.480931 10095 raft_consensus.cc:493] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:13.481263 10095 raft_consensus.cc:3060] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:13.482116 10013 ts_tablet_manager.cc:1434] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260504 13:56:13.482860 10013 tablet_bootstrap.cc:492] T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:13.484525 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b320a37b29844c7593a12000eaa3da43" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:13.486322 9618 leader_election.cc:343] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:13.491472 10095 raft_consensus.cc:515] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.493665 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b320a37b29844c7593a12000eaa3da43" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
W20260504 13:56:13.494813 9618 leader_election.cc:343] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:13.495307 9931 tablet_bootstrap.cc:654] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:13.499742 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b320a37b29844c7593a12000eaa3da43" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:13.500260 9832 raft_consensus.cc:3060] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:13.500370 9341 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_038807318bcc4110b77de9582f6c42f5" } modify_external_catalogs: true
I20260504 13:56:13.500801 9341 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_038807318bcc4110b77de9582f6c42f5" } modify_external_catalogs: true
I20260504 13:56:13.505767 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_0082bd435ced4fdcaeafe8848d05f9bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:13.506856 9832 raft_consensus.cc:2468] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
W20260504 13:56:13.507432 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0082bd435ced4fdcaeafe8848d05f9bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:13.507864 9620 leader_election.cc:304] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:13.515275 10057 raft_consensus.cc:2804] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:13.515964 10057 raft_consensus.cc:697] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:13.519418 9931 tablet_bootstrap.cc:492] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:13.519862 9931 ts_tablet_manager.cc:1403] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.059s user 0.010s sys 0.000s
I20260504 13:56:13.520105 10003 ts_tablet_manager.cc:1929] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:13.519829 10057 consensus_queue.cc:237] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.520493 10003 log.cc:1199] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/03818e34ac0545be880c873cf82cd4ea
I20260504 13:56:13.522776 10003 ts_tablet_manager.cc:1950] T 03818e34ac0545be880c873cf82cd4ea P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:13.522888 9932 tablet_bootstrap.cc:492] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:13.521449 10013 tablet_bootstrap.cc:654] T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:13.523272 9932 ts_tablet_manager.cc:1403] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.070s user 0.010s sys 0.000s
I20260504 13:56:13.525583 9932 raft_consensus.cc:359] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.526163 9932 raft_consensus.cc:385] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.526283 10087 raft_consensus.cc:493] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:13.526443 9932 raft_consensus.cc:740] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:13.526691 10087 raft_consensus.cc:515] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.515266 10095 leader_election.cc:290] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:13.528410 10087 leader_election.cc:290] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:13.532723 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03818e34ac0545be880c873cf82cd4ea" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:13.533223 9539 raft_consensus.cc:2468] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:13.534253 9618 leader_election.cc:304] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:13.535017 10087 raft_consensus.cc:2804] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:13.535390 10087 raft_consensus.cc:493] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:13.535746 10087 raft_consensus.cc:3060] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:13.539407 9932 consensus_queue.cc:260] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.541332 9932 ts_tablet_manager.cc:1434] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.018s user 0.004s sys 0.001s
I20260504 13:56:13.542114 9932 tablet_bootstrap.cc:492] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:13.543659 9620 leader_election.cc:343] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Not found: Tablet not found: 03818e34ac0545be880c873cf82cd4ea
I20260504 13:56:13.547641 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 03818e34ac0545be880c873cf82cd4ea (table default.Table_078d1859743841e985b4e0e1c4f61030 [id=89a6d18328c04d90828f3e51696592fa]) successfully deleted
W20260504 13:56:13.553354 9338 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ed4f5ea230a413a8dcdbb661ce173b6 consensus state has no leader
W20260504 13:56:13.554621 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 69ed77782cd34274932957106ce26766 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69ed77782cd34274932957106ce26766 already in progress: creating tablet
I20260504 13:56:13.554953 10087 raft_consensus.cc:515] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.557503 9932 tablet_bootstrap.cc:654] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:13.559731 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03818e34ac0545be880c873cf82cd4ea" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:13.560230 9539 raft_consensus.cc:3060] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:13.578822 9662 tablet_service.cc:1511] Processing CreateTablet for tablet baf07a970a7c4a35b3c849b06d6c88d0 (DEFAULT_TABLE table=default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.580273 9662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baf07a970a7c4a35b3c849b06d6c88d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.585458 9520 tablet_service.cc:1511] Processing CreateTablet for tablet baf07a970a7c4a35b3c849b06d6c88d0 (DEFAULT_TABLE table=default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.586707 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baf07a970a7c4a35b3c849b06d6c88d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.592694 9517 tablet_service.cc:1511] Processing CreateTablet for tablet 48b193e958b4414aad29dd2bcee1a61b (DEFAULT_TABLE table=default.Table_21845a95c3ab4388b869666380989ec3 [id=e9e56499fd2243919308927b3614ae8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.594197 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b193e958b4414aad29dd2bcee1a61b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.597724 9343 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 52ff55bef5e54353ab9ca09130ab1e0e
I20260504 13:56:13.600941 9516 tablet_service.cc:1558] Processing DeleteTablet for tablet 52ff55bef5e54353ab9ca09130ab1e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:13.601444 9665 tablet_service.cc:1558] Processing DeleteTablet for tablet 52ff55bef5e54353ab9ca09130ab1e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:13.602355 10002 tablet_replica.cc:333] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:13.603287 10002 raft_consensus.cc:2243] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:13.604272 10002 raft_consensus.cc:2272] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:13.609064 9665 tablet_service.cc:1511] Processing CreateTablet for tablet 48b193e958b4414aad29dd2bcee1a61b (DEFAULT_TABLE table=default.Table_21845a95c3ab4388b869666380989ec3 [id=e9e56499fd2243919308927b3614ae8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.610288 10002 ts_tablet_manager.cc:1916] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:13.614481 9341 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 65052a5c28c147a385144317e21eea41
I20260504 13:56:13.617074 9944 tablet_replica.cc:333] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:13.617918 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b193e958b4414aad29dd2bcee1a61b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.621146 9666 tablet_service.cc:1558] Processing DeleteTablet for tablet 65052a5c28c147a385144317e21eea41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:13.623454 9343 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_5fa7db742ec342c7846f99587269c492"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:13.624862 9343 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fa7db742ec342c7846f99587269c492 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:13.630465 9513 tablet_service.cc:1558] Processing DeleteTablet for tablet 65052a5c28c147a385144317e21eea41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:13.631989 9516 tablet_service.cc:1511] Processing CreateTablet for tablet fe7ab3b577b74a9c9a40cb15c94f8aa1 (DEFAULT_TABLE table=default.Table_4b58a5b3e40c435b91e8fb70062833c1 [id=1714df2b31ff4865b6b357500d121a26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.633327 9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe7ab3b577b74a9c9a40cb15c94f8aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.634835 9341 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_21845a95c3ab4388b869666380989ec3" } new_table_name: "default.Table_2c39b14515bb46f6a0b0f1252d8e273d" modify_external_catalogs: true
I20260504 13:56:13.638631 9944 raft_consensus.cc:2243] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:13.639086 9944 raft_consensus.cc:2272] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:13.640838 9944 ts_tablet_manager.cc:1916] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:13.648191 9539 raft_consensus.cc:2468] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:13.653028 9618 leader_election.cc:304] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:13.655128 9335 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 0867df9ea115418193b6d9d3fc251923
I20260504 13:56:13.657689 9666 tablet_service.cc:1558] Processing DeleteTablet for tablet 0867df9ea115418193b6d9d3fc251923 with delete_type TABLET_DATA_DELETED (replaced by tablet c34877a81d504f46a2f680c5d5ddb4f4) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:13.659880 9513 tablet_service.cc:1558] Processing DeleteTablet for tablet 0867df9ea115418193b6d9d3fc251923 with delete_type TABLET_DATA_DELETED (replaced by tablet c34877a81d504f46a2f680c5d5ddb4f4) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:13.660382 9335 catalog_manager.cc:6741] ReplaceTablet: tablet 0867df9ea115418193b6d9d3fc251923 deleted and replaced by tablet c34877a81d504f46a2f680c5d5ddb4f4
I20260504 13:56:13.669251 9666 tablet_service.cc:1511] Processing CreateTablet for tablet fe7ab3b577b74a9c9a40cb15c94f8aa1 (DEFAULT_TABLE table=default.Table_4b58a5b3e40c435b91e8fb70062833c1 [id=1714df2b31ff4865b6b357500d121a26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.670336 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe7ab3b577b74a9c9a40cb15c94f8aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.675602 10057 raft_consensus.cc:2804] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:13.676388 10057 raft_consensus.cc:697] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:13.677116 10057 consensus_queue.cc:237] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.682108 9340 catalog_manager.cc:5671] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:13.688071 9340 catalog_manager.cc:5671] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:13.697327 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_434bdc9867ad475a9719ba0c3760ef71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:13.700450 9336 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_2717e0b44dd14ed1b25e45e06bc0ae88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:13.556831 9931 raft_consensus.cc:359] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.583401 9812 tablet_service.cc:1511] Processing CreateTablet for tablet baf07a970a7c4a35b3c849b06d6c88d0 (DEFAULT_TABLE table=default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.703609 9812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baf07a970a7c4a35b3c849b06d6c88d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.600275 9814 tablet_service.cc:1558] Processing DeleteTablet for tablet 52ff55bef5e54353ab9ca09130ab1e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:13.604249 9811 tablet_service.cc:1511] Processing CreateTablet for tablet 48b193e958b4414aad29dd2bcee1a61b (DEFAULT_TABLE table=default.Table_21845a95c3ab4388b869666380989ec3 [id=e9e56499fd2243919308927b3614ae8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.708060 9811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b193e958b4414aad29dd2bcee1a61b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.706714 9339 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 67f26d71286c4f90abd681f9fc9d1e30
I20260504 13:56:13.619870 9813 tablet_service.cc:1558] Processing DeleteTablet for tablet 65052a5c28c147a385144317e21eea41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:13.654455 9810 tablet_service.cc:1558] Processing DeleteTablet for tablet 0867df9ea115418193b6d9d3fc251923 with delete_type TABLET_DATA_DELETED (replaced by tablet c34877a81d504f46a2f680c5d5ddb4f4) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:13.659240 9809 tablet_service.cc:1558] Processing DeleteTablet for tablet 0867df9ea115418193b6d9d3fc251923 with delete_type TABLET_DATA_DELETED (replaced by tablet c34877a81d504f46a2f680c5d5ddb4f4) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:13.712024 9513 tablet_service.cc:1558] Processing DeleteTablet for tablet 67f26d71286c4f90abd681f9fc9d1e30 with delete_type TABLET_DATA_DELETED (replaced by tablet b8e4239942f34fb081a026a35740e465) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:13.702059 9340 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 0b7a71dbcc694e1a97019188cfb2c672 (table default.Table_6aecd26cb4c04baea5252f5dfbb75663 [id=0649fc4627314bca8b51d0e1f5022e6d]). Expected version 3 got 0
I20260504 13:56:13.721577 10091 raft_consensus.cc:493] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:13.722011 10091 raft_consensus.cc:515] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.723830 10091 leader_election.cc:290] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:13.727439 10003 tablet_replica.cc:333] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:13.728171 10003 raft_consensus.cc:2243] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:13.728696 10003 raft_consensus.cc:2272] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:13.731386 10003 ts_tablet_manager.cc:1916] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:13.733299 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e3b36e1f654a3a8df51480801dabd7" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:13.734551 9767 leader_election.cc:343] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:13.741945 9931 raft_consensus.cc:385] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.742234 9931 raft_consensus.cc:740] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:13.742784 9931 consensus_queue.cc:260] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.744719 9931 ts_tablet_manager.cc:1434] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.225s user 0.004s sys 0.000s
I20260504 13:56:13.758153 9809 tablet_service.cc:1511] Processing CreateTablet for tablet fe7ab3b577b74a9c9a40cb15c94f8aa1 (DEFAULT_TABLE table=default.Table_4b58a5b3e40c435b91e8fb70062833c1 [id=1714df2b31ff4865b6b357500d121a26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.758212 9338 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_d01791667b8a4b77855b70cbed601edb" } new_table_name: "default.Table_ec713dc211a9418f9f9773e25ea10095" modify_external_catalogs: true
I20260504 13:56:13.759346 9809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe7ab3b577b74a9c9a40cb15c94f8aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.759514 9339 catalog_manager.cc:6741] ReplaceTablet: tablet 67f26d71286c4f90abd681f9fc9d1e30 deleted and replaced by tablet b8e4239942f34fb081a026a35740e465
W20260504 13:56:13.760568 9336 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2717e0b44dd14ed1b25e45e06bc0ae88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:13.763020 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_434bdc9867ad475a9719ba0c3760ef71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:13.764710 9513 tablet_service.cc:1511] Processing CreateTablet for tablet cdb86f6370bf4c5f9c488d85efb67ef7 (DEFAULT_TABLE table=default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.764962 9813 tablet_service.cc:1558] Processing DeleteTablet for tablet 67f26d71286c4f90abd681f9fc9d1e30 with delete_type TABLET_DATA_DELETED (replaced by tablet b8e4239942f34fb081a026a35740e465) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:13.766309 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb86f6370bf4c5f9c488d85efb67ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.767897 9810 tablet_service.cc:1511] Processing CreateTablet for tablet cdb86f6370bf4c5f9c488d85efb67ef7 (DEFAULT_TABLE table=default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.768658 9514 tablet_service.cc:1511] Processing CreateTablet for tablet 65052a5c28c147a385144317e21eea41 (DEFAULT_TABLE table=default.Table_038807318bcc4110b77de9582f6c42f5 [id=c19c0f3f26934ad6bbf2d4202cdaa542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.768983 9810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb86f6370bf4c5f9c488d85efb67ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.769639 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65052a5c28c147a385144317e21eea41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.780057 9813 tablet_service.cc:1511] Processing CreateTablet for tablet 65052a5c28c147a385144317e21eea41 (DEFAULT_TABLE table=default.Table_038807318bcc4110b77de9582f6c42f5 [id=c19c0f3f26934ad6bbf2d4202cdaa542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.781015 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65052a5c28c147a385144317e21eea41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.783044 9814 tablet_service.cc:1511] Processing CreateTablet for tablet afee197774db411d97468913a207703a (DEFAULT_TABLE table=default.Table_3bed4e09fe9d4eceb3bc1614c4cc26bf [id=a2697dba5f38435c81dc52316712917d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.783922 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afee197774db411d97468913a207703a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.783952 9932 tablet_bootstrap.cc:492] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:13.784355 9932 ts_tablet_manager.cc:1403] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.242s user 0.006s sys 0.011s
I20260504 13:56:13.784281 9515 tablet_service.cc:1511] Processing CreateTablet for tablet 2b50573310a04f3694eeab2fef0fb540 (DEFAULT_TABLE table=default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.785773 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b50573310a04f3694eeab2fef0fb540. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.787240 9932 raft_consensus.cc:359] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.787828 9932 raft_consensus.cc:385] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.788089 9932 raft_consensus.cc:740] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:13.788635 9932 consensus_queue.cc:260] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.790153 10087 leader_election.cc:290] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:13.790953 9932 ts_tablet_manager.cc:1434] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260504 13:56:13.795570 10002 ts_tablet_manager.cc:1929] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:13.795904 10002 log.cc:1199] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/52ff55bef5e54353ab9ca09130ab1e0e
I20260504 13:56:13.796589 10002 ts_tablet_manager.cc:1950] T 52ff55bef5e54353ab9ca09130ab1e0e P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:13.797837 9931 tablet_bootstrap.cc:492] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:13.800091 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 52ff55bef5e54353ab9ca09130ab1e0e (table default.Table_d4a59933eb124b4295cc0b066a46f9ea [id=493e2db02edb42c79460ec701aa92679]) successfully deleted
I20260504 13:56:13.802443 9343 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_47c9ed0b601146b687a576ec89c8b626"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:13.803874 9343 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47c9ed0b601146b687a576ec89c8b626 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:13.809332 9339 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_a0a1002687744b879e47ae4fd701d903" } modify_external_catalogs: true
I20260504 13:56:13.809850 9339 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_a0a1002687744b879e47ae4fd701d903" } modify_external_catalogs: true
I20260504 13:56:13.810457 10013 tablet_bootstrap.cc:492] T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:13.810834 10013 ts_tablet_manager.cc:1403] T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.328s user 0.074s sys 0.111s
I20260504 13:56:13.814522 10013 raft_consensus.cc:359] T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.815088 10013 raft_consensus.cc:385] T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.815369 10013 raft_consensus.cc:740] T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:13.815929 10013 consensus_queue.cc:260] T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.817706 10013 ts_tablet_manager.cc:1434] T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20260504 13:56:13.821593 9808 tablet_service.cc:1511] Processing CreateTablet for tablet 2b50573310a04f3694eeab2fef0fb540 (DEFAULT_TABLE table=default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.824896 9337 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_6aecd26cb4c04baea5252f5dfbb75663" } modify_external_catalogs: true
I20260504 13:56:13.825399 9337 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_6aecd26cb4c04baea5252f5dfbb75663" } modify_external_catalogs: true
I20260504 13:56:13.823082 9808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b50573310a04f3694eeab2fef0fb540. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.827821 10013 tablet_bootstrap.cc:492] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:13.832883 9341 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 48b193e958b4414aad29dd2bcee1a61b consensus state has no leader
I20260504 13:56:13.838538 10013 tablet_bootstrap.cc:654] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:13.847919 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 65052a5c28c147a385144317e21eea41 because the tablet was not found. No further retry: Not found: Tablet not found: 65052a5c28c147a385144317e21eea41
I20260504 13:56:13.843947 10002 tablet_replica.cc:333] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
W20260504 13:56:13.851543 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 0867df9ea115418193b6d9d3fc251923 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0867df9ea115418193b6d9d3fc251923 already in progress: creating tablet
I20260504 13:56:13.851928 10002 raft_consensus.cc:2243] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:13.852674 10002 pending_rounds.cc:70] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00: Trying to abort 1 pending ops.
I20260504 13:56:13.852967 10002 pending_rounds.cc:77] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290577665441792 op_type: NO_OP noop_request { }
I20260504 13:56:13.853382 10002 raft_consensus.cc:2889] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:13.853807 10002 raft_consensus.cc:2272] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:13.856258 9517 tablet_service.cc:1511] Processing CreateTablet for tablet afee197774db411d97468913a207703a (DEFAULT_TABLE table=default.Table_3bed4e09fe9d4eceb3bc1614c4cc26bf [id=a2697dba5f38435c81dc52316712917d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.859928 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afee197774db411d97468913a207703a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.861487 10013 tablet_bootstrap.cc:492] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:13.871326 10013 ts_tablet_manager.cc:1403] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.004s
I20260504 13:56:13.873509 10013 raft_consensus.cc:359] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.874069 10013 raft_consensus.cc:385] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.874327 10013 raft_consensus.cc:740] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:13.848434 9931 tablet_bootstrap.cc:654] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:13.877938 10013 consensus_queue.cc:260] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.879662 10013 ts_tablet_manager.cc:1434] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.008s user 0.003s sys 0.001s
I20260504 13:56:13.880390 10013 tablet_bootstrap.cc:492] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:13.881047 10002 ts_tablet_manager.cc:1916] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:13.886207 9341 master_service.cc:954] ReplaceTablet: received request to replace tablet cdb86f6370bf4c5f9c488d85efb67ef7 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:13.903517 10013 tablet_bootstrap.cc:654] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:13.926726 9335 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet e0e3b36e1f654a3a8df51480801dabd7 (table default.Table_9e9ef3353dcf4b1fadaa2f51bc8f7041 [id=8dcabd7d67334598926f927a62e630d6]). Expected version 2 got 0
I20260504 13:56:13.929459 10013 tablet_bootstrap.cc:492] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:13.929845 10013 ts_tablet_manager.cc:1403] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.003s
I20260504 13:56:13.930919 9344 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: default.Table_15bf9089e68d4882838fe4f1c65156ef [id=3814e135130a4213a46df3cd93eca49e] alter complete (version 1)
I20260504 13:56:13.932334 10013 raft_consensus.cc:359] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.932904 10013 raft_consensus.cc:385] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:13.933136 10013 raft_consensus.cc:740] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:13.933677 10013 consensus_queue.cc:260] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.936704 9517 tablet_service.cc:1558] Processing DeleteTablet for tablet 03818e34ac0545be880c873cf82cd4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:13.939309 9513 tablet_service.cc:1558] Processing DeleteTablet for tablet 52ff55bef5e54353ab9ca09130ab1e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:52642
W20260504 13:56:13.942239 9330 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet afee197774db411d97468913a207703a (table default.Table_3bed4e09fe9d4eceb3bc1614c4cc26bf [id=a2697dba5f38435c81dc52316712917d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet afee197774db411d97468913a207703a consensus state has no leader
I20260504 13:56:13.948575 9812 tablet_service.cc:1558] Processing DeleteTablet for tablet d7e556f81ec940f1aa3c55b3a25f31ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:13.949877 10013 ts_tablet_manager.cc:1434] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260504 13:56:13.950716 10013 tablet_bootstrap.cc:492] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:13.968827 9932 tablet_bootstrap.cc:492] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:13.968943 9944 ts_tablet_manager.cc:1929] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:13.970711 9944 log.cc:1199] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/52ff55bef5e54353ab9ca09130ab1e0e
I20260504 13:56:13.971774 9944 ts_tablet_manager.cc:1950] T 52ff55bef5e54353ab9ca09130ab1e0e P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:13.975250 10057 raft_consensus.cc:493] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:13.975924 10057 raft_consensus.cc:515] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:13.977041 9932 tablet_bootstrap.cc:654] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:13.906337 9620 leader_election.cc:343] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Not found: Tablet not found: 03818e34ac0545be880c873cf82cd4ea
W20260504 13:56:13.977788 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 71654.5us
I20260504 13:56:13.911615 9660 tablet_service.cc:1511] Processing CreateTablet for tablet 2b50573310a04f3694eeab2fef0fb540 (DEFAULT_TABLE table=default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.979189 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b50573310a04f3694eeab2fef0fb540. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.984155 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 52ff55bef5e54353ab9ca09130ab1e0e (table default.Table_d4a59933eb124b4295cc0b066a46f9ea [id=493e2db02edb42c79460ec701aa92679]) successfully deleted
I20260504 13:56:13.985229 10013 tablet_bootstrap.cc:654] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:13.913035 9661 tablet_service.cc:1511] Processing CreateTablet for tablet cdb86f6370bf4c5f9c488d85efb67ef7 (DEFAULT_TABLE table=default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.989739 9661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb86f6370bf4c5f9c488d85efb67ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.991449 9344 master_service.cc:954] ReplaceTablet: received request to replace tablet b4dca0cadd5e400eab1365c03a8de4f4 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:13.914589 9663 tablet_service.cc:1511] Processing CreateTablet for tablet afee197774db411d97468913a207703a (DEFAULT_TABLE table=default.Table_3bed4e09fe9d4eceb3bc1614c4cc26bf [id=a2697dba5f38435c81dc52316712917d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.995932 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afee197774db411d97468913a207703a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.997706 9339 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 14b7f1e834d0439c89abb180bd13853d
I20260504 13:56:13.998659 10013 tablet_bootstrap.cc:492] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:13.999006 10013 ts_tablet_manager.cc:1403] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.000s
I20260504 13:56:13.916239 9664 tablet_service.cc:1511] Processing CreateTablet for tablet 65052a5c28c147a385144317e21eea41 (DEFAULT_TABLE table=default.Table_038807318bcc4110b77de9582f6c42f5 [id=c19c0f3f26934ad6bbf2d4202cdaa542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:13.949560 9666 tablet_service.cc:1558] Processing DeleteTablet for tablet 67f26d71286c4f90abd681f9fc9d1e30 with delete_type TABLET_DATA_DELETED (replaced by tablet b8e4239942f34fb081a026a35740e465) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:13.999881 9664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65052a5c28c147a385144317e21eea41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:13.951854 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e3b36e1f654a3a8df51480801dabd7" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:14.004981 9341 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet cdb86f6370bf4c5f9c488d85efb67ef7
I20260504 13:56:14.006150 9341 catalog_manager.cc:6741] ReplaceTablet: tablet cdb86f6370bf4c5f9c488d85efb67ef7 deleted and replaced by tablet 030b35c3409143108c34ab1f0ee1b96d
I20260504 13:56:14.007673 9517 tablet_service.cc:1558] Processing DeleteTablet for tablet cdb86f6370bf4c5f9c488d85efb67ef7 with delete_type TABLET_DATA_DELETED (replaced by tablet 030b35c3409143108c34ab1f0ee1b96d) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:14.010372 9814 tablet_service.cc:1558] Processing DeleteTablet for tablet 14b7f1e834d0439c89abb180bd13853d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:14.012786 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_af058f50341e4f478c970f8cb52bf0d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:14.014744 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af058f50341e4f478c970f8cb52bf0d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:14.014655 10013 raft_consensus.cc:359] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.015250 10013 raft_consensus.cc:385] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:14.015591 10013 raft_consensus.cc:740] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:14.016323 10013 consensus_queue.cc:260] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.016839 9931 tablet_bootstrap.cc:492] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:14.018133 10013 ts_tablet_manager.cc:1434] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20260504 13:56:14.018900 10013 tablet_bootstrap.cc:492] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:14.027894 9768 leader_election.cc:343] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:14.028020 9666 tablet_service.cc:1558] Processing DeleteTablet for tablet 14b7f1e834d0439c89abb180bd13853d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:14.028319 9768 leader_election.cc:304] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:14.029330 10091 raft_consensus.cc:2749] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:14.035437 9513 tablet_service.cc:1558] Processing DeleteTablet for tablet 14b7f1e834d0439c89abb180bd13853d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:14.037528 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91371bf848f5465ea2db0c090583382d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:14.038061 9832 raft_consensus.cc:2468] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:14.039134 9620 leader_election.cc:304] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:14.039274 10057 leader_election.cc:290] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:14.041105 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91371bf848f5465ea2db0c090583382d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:14.041613 10057 raft_consensus.cc:493] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:14.041776 9540 raft_consensus.cc:2468] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:14.041976 10057 raft_consensus.cc:515] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.044595 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fdc09a498be470ab04563c62caf7755" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:14.045120 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fdc09a498be470ab04563c62caf7755" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:14.045980 9618 leader_election.cc:343] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:14.046602 9620 leader_election.cc:343] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:14.047036 9620 leader_election.cc:304] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:14.042222 9341 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_c9e39aac400445028318ffebeb95209d" } new_table_name: "default.Table_f4a4640b6bb0432dad74376a7ed31181" modify_external_catalogs: true
I20260504 13:56:14.055799 9810 tablet_service.cc:1558] Processing DeleteTablet for tablet cdb86f6370bf4c5f9c488d85efb67ef7 with delete_type TABLET_DATA_DELETED (replaced by tablet 030b35c3409143108c34ab1f0ee1b96d) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:14.065094 9665 tablet_service.cc:1558] Processing DeleteTablet for tablet cdb86f6370bf4c5f9c488d85efb67ef7 with delete_type TABLET_DATA_DELETED (replaced by tablet 030b35c3409143108c34ab1f0ee1b96d) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:14.070968 10075 raft_consensus.cc:493] T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:14.071487 10075 raft_consensus.cc:515] T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.073205 10075 leader_election.cc:290] T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:14.074620 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b320a37b29844c7593a12000eaa3da43" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:14.074927 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b320a37b29844c7593a12000eaa3da43" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:14.075673 9832 raft_consensus.cc:2393] T b320a37b29844c7593a12000eaa3da43 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 1: Already voted for candidate 5beceb6317444247b31b44a6581707c6 in this term.
I20260504 13:56:14.071607 10057 leader_election.cc:290] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:14.076521 9474 leader_election.cc:304] T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:14.077569 10075 raft_consensus.cc:3060] T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.081317 9931 ts_tablet_manager.cc:1403] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.284s user 0.010s sys 0.000s
I20260504 13:56:14.078974 10057 raft_consensus.cc:2804] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:14.083657 10057 raft_consensus.cc:493] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:14.084350 10087 raft_consensus.cc:2749] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:14.085465 10075 raft_consensus.cc:2749] T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:14.086778 10003 ts_tablet_manager.cc:1929] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:14.087117 10003 log.cc:1199] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/52ff55bef5e54353ab9ca09130ab1e0e
I20260504 13:56:14.090835 10091 raft_consensus.cc:493] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:14.093127 10091 raft_consensus.cc:515] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.094951 10091 leader_election.cc:290] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:14.096022 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdf1c3cef3334a849f7b5e429b7f4990" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:14.097558 9767 leader_election.cc:343] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:14.096374 10003 ts_tablet_manager.cc:1950] T 52ff55bef5e54353ab9ca09130ab1e0e P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:14.091547 10057 raft_consensus.cc:3060] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.102819 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdf1c3cef3334a849f7b5e429b7f4990" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:14.103713 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 52ff55bef5e54353ab9ca09130ab1e0e (table default.Table_d4a59933eb124b4295cc0b066a46f9ea [id=493e2db02edb42c79460ec701aa92679]) successfully deleted
W20260504 13:56:14.104132 9768 leader_election.cc:343] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:14.104563 9768 leader_election.cc:304] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:14.105585 10091 raft_consensus.cc:2749] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:14.110168 9344 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet b4dca0cadd5e400eab1365c03a8de4f4
I20260504 13:56:14.111492 9344 catalog_manager.cc:6741] ReplaceTablet: tablet b4dca0cadd5e400eab1365c03a8de4f4 deleted and replaced by tablet 604f1ba5f71041a7877daff8972a994a
I20260504 13:56:14.114367 9513 tablet_service.cc:1558] Processing DeleteTablet for tablet b4dca0cadd5e400eab1365c03a8de4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 604f1ba5f71041a7877daff8972a994a) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:14.115234 9810 tablet_service.cc:1558] Processing DeleteTablet for tablet b4dca0cadd5e400eab1365c03a8de4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 604f1ba5f71041a7877daff8972a994a) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:14.128533 9344 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_443402c40ae5443daf4ba4258b847303"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:14.128818 10127 tablet.cc:1724] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20260504 13:56:14.130119 9344 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_443402c40ae5443daf4ba4258b847303 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:14.137908 9328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260504 13:56:14.151419 9665 tablet_service.cc:1558] Processing DeleteTablet for tablet b4dca0cadd5e400eab1365c03a8de4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 604f1ba5f71041a7877daff8972a994a) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:14.153803 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 65052a5c28c147a385144317e21eea41 because the tablet was not found. No further retry: Not found: Tablet not found: 65052a5c28c147a385144317e21eea41
I20260504 13:56:14.163987 9944 tablet_replica.cc:333] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:14.164764 9944 raft_consensus.cc:2243] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:14.165212 9944 raft_consensus.cc:2272] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:14.166947 9944 ts_tablet_manager.cc:1916] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260504 13:56:14.173413 9329 catalog_manager.cc:4982] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 0867df9ea115418193b6d9d3fc251923 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0867df9ea115418193b6d9d3fc251923 already in progress: creating tablet
I20260504 13:56:14.182164 9539 raft_consensus.cc:1275] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Refusing update from remote peer 5beceb6317444247b31b44a6581707c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 13:56:14.183768 10095 consensus_queue.cc:1048] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:14.189378 10057 raft_consensus.cc:515] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.194480 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91371bf848f5465ea2db0c090583382d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:14.194984 9540 raft_consensus.cc:3060] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.202569 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91371bf848f5465ea2db0c090583382d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:14.191140 10057 leader_election.cc:290] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:14.203291 9834 raft_consensus.cc:3060] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.204625 10075 raft_consensus.cc:493] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:14.205046 10075 raft_consensus.cc:515] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.206768 10075 leader_election.cc:290] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:14.210614 9834 raft_consensus.cc:2468] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:14.210829 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b7a71dbcc694e1a97019188cfb2c672" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:14.211503 9684 raft_consensus.cc:2468] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:14.211853 9620 leader_election.cc:304] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:14.213160 9473 leader_election.cc:304] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:14.213577 10095 raft_consensus.cc:2804] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:14.214406 9540 raft_consensus.cc:2468] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:14.215408 10095 raft_consensus.cc:697] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:14.215631 10075 raft_consensus.cc:2804] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:14.215948 10075 raft_consensus.cc:493] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:14.216238 10075 raft_consensus.cc:3060] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.216226 10095 consensus_queue.cc:237] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.220975 9932 tablet_bootstrap.cc:492] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:14.221365 9932 ts_tablet_manager.cc:1403] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.255s user 0.027s sys 0.014s
I20260504 13:56:14.237516 9944 ts_tablet_manager.cc:1929] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:14.225917 10075 raft_consensus.cc:515] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.242934 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b7a71dbcc694e1a97019188cfb2c672" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:14.243642 9832 raft_consensus.cc:1275] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Refusing update from remote peer 5beceb6317444247b31b44a6581707c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 13:56:14.243620 9833 raft_consensus.cc:2468] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:14.247866 9335 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_cdee25533fe54da0a74b421714b0d7c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:14.248773 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b7a71dbcc694e1a97019188cfb2c672" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:14.249325 9832 raft_consensus.cc:3060] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.256594 10075 leader_election.cc:290] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:14.260877 9832 raft_consensus.cc:2468] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
W20260504 13:56:14.264132 9328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 126227us
I20260504 13:56:14.266536 9474 leader_election.cc:304] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:14.277107 10133 raft_consensus.cc:2804] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:14.279536 9931 raft_consensus.cc:359] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.282897 10133 raft_consensus.cc:697] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:14.240883 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b7a71dbcc694e1a97019188cfb2c672" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:14.294917 9944 log.cc:1199] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/67f26d71286c4f90abd681f9fc9d1e30
I20260504 13:56:14.295022 9684 raft_consensus.cc:3060] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.296362 9944 ts_tablet_manager.cc:1950] T 67f26d71286c4f90abd681f9fc9d1e30 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:14.288420 10075 raft_consensus.cc:493] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:14.297492 10075 raft_consensus.cc:515] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.298821 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 67f26d71286c4f90abd681f9fc9d1e30 (table default.Table_895e961d6c1a45e688ad4e0ca752bae9 [id=73e12524eb9e416782bf9f7fb58c93bd]) successfully deleted
I20260504 13:56:14.299800 10075 leader_election.cc:290] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:14.299818 9344 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_5fa7db742ec342c7846f99587269c492" } new_table_name: "default.Table_1f5a66e30f35482dab7c556daebc87c9" modify_external_catalogs: true
I20260504 13:56:14.245383 10057 consensus_queue.cc:1048] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:14.303707 10133 consensus_queue.cc:237] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.304275 9684 raft_consensus.cc:2468] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:14.307157 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e556f81ec940f1aa3c55b3a25f31ab" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:14.308317 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_737b85d6e3e04656a60981e69282cd7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:14.309865 9339 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_0082bd435ced4fdcaeafe8848d05f9bf" } modify_external_catalogs: true
I20260504 13:56:14.310287 9339 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_0082bd435ced4fdcaeafe8848d05f9bf" } modify_external_catalogs: true
I20260504 13:56:14.310808 10013 tablet_bootstrap.cc:654] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:14.312976 9473 leader_election.cc:343] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:14.256994 10132 tablet.cc:1724] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:14.262414 9932 raft_consensus.cc:359] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.319363 9932 raft_consensus.cc:385] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:14.319762 9932 raft_consensus.cc:740] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:14.320461 9932 consensus_queue.cc:260] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.326134 9932 ts_tablet_manager.cc:1434] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.104s user 0.006s sys 0.001s
I20260504 13:56:14.301443 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e556f81ec940f1aa3c55b3a25f31ab" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:14.302475 10091 raft_consensus.cc:493] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:14.327070 10091 raft_consensus.cc:515] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.329376 10091 leader_election.cc:290] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:14.330919 9832 raft_consensus.cc:2468] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:14.334306 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e556f81ec940f1aa3c55b3a25f31ab" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:14.334564 10137 tablet.cc:1724] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:14.336267 9474 leader_election.cc:304] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:14.336591 9931 raft_consensus.cc:385] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:14.334940 9540 raft_consensus.cc:2468] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:14.337064 9931 raft_consensus.cc:740] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:14.338347 10075 raft_consensus.cc:2804] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:14.338729 10075 raft_consensus.cc:493] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:14.338099 9931 consensus_queue.cc:260] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.339321 10075 raft_consensus.cc:3060] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.339497 9767 leader_election.cc:304] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:14.342962 10126 tablet.cc:1724] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260504 13:56:14.344959 10141 raft_consensus.cc:2804] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:14.345292 10141 raft_consensus.cc:493] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:14.345626 10141 raft_consensus.cc:3060] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.346508 10003 tablet_replica.cc:333] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
W20260504 13:56:14.347645 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 65052a5c28c147a385144317e21eea41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65052a5c28c147a385144317e21eea41 already in progress: creating tablet
W20260504 13:56:14.308322 9328 consensus_peers.cc:597] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 -> Peer 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337): Couldn't send request to peer 6caedf407d13459e8fd477e8b60b7dc7. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260504 13:56:14.335759 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e556f81ec940f1aa3c55b3a25f31ab" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:14.350623 9932 tablet_bootstrap.cc:492] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:14.352002 9768 leader_election.cc:343] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:14.285188 10136 tablet.cc:1724] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260504 13:56:14.346809 10075 raft_consensus.cc:515] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.359104 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e556f81ec940f1aa3c55b3a25f31ab" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
W20260504 13:56:14.360239 9473 leader_election.cc:343] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:14.361166 10075 leader_election.cc:290] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
W20260504 13:56:14.362226 9329 catalog_manager.cc:4982] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 14b7f1e834d0439c89abb180bd13853d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14b7f1e834d0439c89abb180bd13853d already in progress: creating tablet
I20260504 13:56:14.362586 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e556f81ec940f1aa3c55b3a25f31ab" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:14.363274 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_737b85d6e3e04656a60981e69282cd7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:14.368127 10013 tablet_bootstrap.cc:492] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:14.368525 10013 ts_tablet_manager.cc:1403] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.350s user 0.034s sys 0.055s
W20260504 13:56:14.370262 9335 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdee25533fe54da0a74b421714b0d7c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:14.370764 10013 raft_consensus.cc:359] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.371505 10013 raft_consensus.cc:385] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:14.371754 10013 raft_consensus.cc:740] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:14.372346 10013 consensus_queue.cc:260] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.374173 10013 ts_tablet_manager.cc:1434] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260504 13:56:14.376785 10013 tablet_bootstrap.cc:492] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:14.374354 9328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 66049us
W20260504 13:56:14.385392 9329 catalog_manager.cc:4982] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet cdb86f6370bf4c5f9c488d85efb67ef7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cdb86f6370bf4c5f9c488d85efb67ef7 already in progress: creating tablet
I20260504 13:56:14.387037 9944 tablet_replica.cc:333] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:14.387915 9944 raft_consensus.cc:2243] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:14.388907 9944 raft_consensus.cc:2272] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:14.390661 9944 ts_tablet_manager.cc:1916] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:14.396209 10013 tablet_bootstrap.cc:654] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:14.406672 9344 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c43ff1d759a40d68344f25f066416ac has no consensus state
I20260504 13:56:14.408991 10013 tablet_bootstrap.cc:492] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:14.409363 10013 ts_tablet_manager.cc:1403] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.004s
I20260504 13:56:14.409725 10141 raft_consensus.cc:515] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.411484 9832 raft_consensus.cc:2393] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 1: Already voted for candidate d4051d2bf0bf410eb8acd107560df888 in this term.
I20260504 13:56:14.412923 9474 leader_election.cc:304] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:14.413946 10075 raft_consensus.cc:2749] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260504 13:56:14.421976 10141 leader_election.cc:290] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:14.422137 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e556f81ec940f1aa3c55b3a25f31ab" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:14.422801 9540 raft_consensus.cc:2393] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:14.425721 10013 raft_consensus.cc:359] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.403821 9932 tablet_bootstrap.cc:654] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:14.416410 10087 raft_consensus.cc:493] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:14.429136 10087 raft_consensus.cc:515] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.430486 10013 raft_consensus.cc:385] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:14.430723 10013 raft_consensus.cc:740] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:14.430821 10087 leader_election.cc:290] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:14.431269 10013 consensus_queue.cc:260] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.432163 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867df9ea115418193b6d9d3fc251923" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:14.432655 9540 raft_consensus.cc:2468] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:14.434270 9618 leader_election.cc:304] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:14.421964 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e556f81ec940f1aa3c55b3a25f31ab" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:14.434742 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867df9ea115418193b6d9d3fc251923" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:14.435299 9832 raft_consensus.cc:2468] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:14.435516 10087 raft_consensus.cc:2804] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:14.435853 10087 raft_consensus.cc:493] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:14.436179 10087 raft_consensus.cc:3060] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.438416 10013 ts_tablet_manager.cc:1434] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.029s user 0.005s sys 0.001s
W20260504 13:56:14.440843 9768 leader_election.cc:343] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:14.441226 9768 leader_election.cc:304] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:14.441875 10141 raft_consensus.cc:2749] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260504 13:56:14.439221 10013 tablet_bootstrap.cc:492] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:14.451778 10087 raft_consensus.cc:515] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.454293 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867df9ea115418193b6d9d3fc251923" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:14.454790 9540 raft_consensus.cc:3060] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.456786 10013 tablet_bootstrap.cc:654] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:14.460906 9540 raft_consensus.cc:2468] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:14.461905 9618 leader_election.cc:304] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:14.462598 10057 raft_consensus.cc:2804] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:14.463045 10057 raft_consensus.cc:697] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:14.464048 10057 consensus_queue.cc:237] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.468041 9932 tablet_bootstrap.cc:492] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:14.472730 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0867df9ea115418193b6d9d3fc251923" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:14.473263 9832 raft_consensus.cc:3060] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.474516 9932 ts_tablet_manager.cc:1403] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.124s user 0.006s sys 0.005s
I20260504 13:56:14.476044 10013 tablet_bootstrap.cc:492] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:14.476544 10013 ts_tablet_manager.cc:1403] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.001s
I20260504 13:56:14.476832 9932 raft_consensus.cc:359] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.477438 9932 raft_consensus.cc:385] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:14.477718 9932 raft_consensus.cc:740] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:14.480077 9832 raft_consensus.cc:2468] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:14.484107 10087 leader_election.cc:290] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:14.486303 10075 raft_consensus.cc:493] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:14.486747 10075 raft_consensus.cc:515] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.487035 10013 raft_consensus.cc:359] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.487833 10013 raft_consensus.cc:385] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:14.488080 10013 raft_consensus.cc:740] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:14.488446 10075 leader_election.cc:290] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:14.489593 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14b7f1e834d0439c89abb180bd13853d" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:14.490022 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14b7f1e834d0439c89abb180bd13853d" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:14.490257 9832 raft_consensus.cc:2468] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:14.491230 9474 leader_election.cc:304] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:14.491423 9932 consensus_queue.cc:260] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:14.491786 9473 leader_election.cc:343] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:14.492596 10075 raft_consensus.cc:2804] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:14.492913 10075 raft_consensus.cc:493] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:14.493690 10075 raft_consensus.cc:3060] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.493881 9932 ts_tablet_manager.cc:1434] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260504 13:56:14.497311 9932 tablet_bootstrap.cc:492] T e0e3b36e1f654a3a8df51480801dabd7 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:14.499981 10013 consensus_queue.cc:260] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.511833 10002 ts_tablet_manager.cc:1929] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:14.520440 9932 tablet_bootstrap.cc:654] T e0e3b36e1f654a3a8df51480801dabd7 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:14.535853 9932 tablet_bootstrap.cc:492] T e0e3b36e1f654a3a8df51480801dabd7 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:14.536230 9932 ts_tablet_manager.cc:1403] T e0e3b36e1f654a3a8df51480801dabd7 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.000s
I20260504 13:56:14.538859 9932 raft_consensus.cc:359] T e0e3b36e1f654a3a8df51480801dabd7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.539613 9932 raft_consensus.cc:385] T e0e3b36e1f654a3a8df51480801dabd7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:14.539870 9932 raft_consensus.cc:740] T e0e3b36e1f654a3a8df51480801dabd7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:14.551421 9932 consensus_queue.cc:260] T e0e3b36e1f654a3a8df51480801dabd7 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.556509 9932 ts_tablet_manager.cc:1434] T e0e3b36e1f654a3a8df51480801dabd7 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260504 13:56:14.557363 9932 tablet_bootstrap.cc:492] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:14.558617 10003 raft_consensus.cc:2243] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:14.559135 10003 raft_consensus.cc:2272] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:14.560981 10003 ts_tablet_manager.cc:1916] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:14.568324 9931 ts_tablet_manager.cc:1434] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.487s user 0.003s sys 0.004s
I20260504 13:56:14.575604 9932 tablet_bootstrap.cc:654] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:14.577044 10087 consensus_queue.cc:1048] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260504 13:56:14.579599 9344 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_a4d8b2884f1e402fa29eee35f1c46d0a" } new_table_name: "default.Table_c5584202645f4ef78e4f6e00aa2bf2c1" modify_external_catalogs: true
I20260504 13:56:14.583052 9343 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_6c0e4aec8a8a4b35a9c5509f3be15772"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:14.584486 9343 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c0e4aec8a8a4b35a9c5509f3be15772 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:14.513453 10143 tablet.cc:1724] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:14.623977 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_6462cb42c34a4eb8976d8d2a4f82d434"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:14.625628 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6462cb42c34a4eb8976d8d2a4f82d434 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:14.634390 9931 tablet_bootstrap.cc:492] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:14.647619 9336 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_3bed4e09fe9d4eceb3bc1614c4cc26bf" } new_table_name: "default.Table_91c2d3ed5c5c42758c87f1f726af88c0" modify_external_catalogs: true
I20260504 13:56:14.512629 10013 ts_tablet_manager.cc:1434] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.036s user 0.009s sys 0.000s
I20260504 13:56:14.671314 10013 tablet_bootstrap.cc:492] T bdf1c3cef3334a849f7b5e429b7f4990 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:14.611621 10075 raft_consensus.cc:515] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.651257 10149 raft_consensus.cc:493] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:14.675860 10075 leader_election.cc:290] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:14.676976 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14b7f1e834d0439c89abb180bd13853d" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:14.677968 9833 raft_consensus.cc:3060] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.677809 10002 log.cc:1199] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/67f26d71286c4f90abd681f9fc9d1e30
I20260504 13:56:14.678929 10002 ts_tablet_manager.cc:1950] T 67f26d71286c4f90abd681f9fc9d1e30 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:14.680195 10002 tablet_replica.cc:333] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:14.680846 10002 raft_consensus.cc:2243] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:14.681233 10002 raft_consensus.cc:2272] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:14.681715 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14b7f1e834d0439c89abb180bd13853d" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:14.674512 10149 raft_consensus.cc:515] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.683262 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 67f26d71286c4f90abd681f9fc9d1e30 (table default.Table_895e961d6c1a45e688ad4e0ca752bae9 [id=73e12524eb9e416782bf9f7fb58c93bd]) successfully deleted
I20260504 13:56:14.683924 10095 consensus_queue.cc:1048] T b320a37b29844c7593a12000eaa3da43 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:14.684363 10149 leader_election.cc:290] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
W20260504 13:56:14.685102 9473 leader_election.cc:343] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:14.685523 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e3b36e1f654a3a8df51480801dabd7" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:14.685716 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e3b36e1f654a3a8df51480801dabd7" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:14.686089 9832 raft_consensus.cc:2468] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:14.686251 9685 raft_consensus.cc:2468] T e0e3b36e1f654a3a8df51480801dabd7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:14.687292 9473 leader_election.cc:304] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:14.687651 10013 tablet_bootstrap.cc:654] T bdf1c3cef3334a849f7b5e429b7f4990 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:14.691340 10149 raft_consensus.cc:2804] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:14.691718 10149 raft_consensus.cc:493] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:14.692080 10149 raft_consensus.cc:3060] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:14.708559 9620 consensus_peers.cc:597] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03818e34ac0545be880c873cf82cd4ea. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:14.720006 10013 tablet_bootstrap.cc:492] T bdf1c3cef3334a849f7b5e429b7f4990 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:14.720403 10013 ts_tablet_manager.cc:1403] T bdf1c3cef3334a849f7b5e429b7f4990 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.009s
I20260504 13:56:14.722532 10013 raft_consensus.cc:359] T bdf1c3cef3334a849f7b5e429b7f4990 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.723097 10013 raft_consensus.cc:385] T bdf1c3cef3334a849f7b5e429b7f4990 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:14.723356 10013 raft_consensus.cc:740] T bdf1c3cef3334a849f7b5e429b7f4990 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:14.723901 10013 consensus_queue.cc:260] T bdf1c3cef3334a849f7b5e429b7f4990 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.725648 10013 ts_tablet_manager.cc:1434] T bdf1c3cef3334a849f7b5e429b7f4990 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260504 13:56:14.726431 10013 tablet_bootstrap.cc:492] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:14.729321 9660 tablet_service.cc:1511] Processing CreateTablet for tablet 3c43ff1d759a40d68344f25f066416ac (DEFAULT_TABLE table=default.Table_1f5a66e30f35482dab7c556daebc87c9 [id=c268fa33d24540b4978e3feebb0f793f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:14.730439 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c43ff1d759a40d68344f25f066416ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:14.731356 9337 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 0b7a71dbcc694e1a97019188cfb2c672
W20260504 13:56:14.737658 9340 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 25) took 1111 ms (client timeout 1000 ms). Trace:
I20260504 13:56:14.739851 9810 tablet_service.cc:1558] Processing DeleteTablet for tablet 0b7a71dbcc694e1a97019188cfb2c672 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:14.740178 10013 tablet_bootstrap.cc:654] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:14.741580 9663 tablet_service.cc:1558] Processing DeleteTablet for tablet 0b7a71dbcc694e1a97019188cfb2c672 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:14.738015 9340 rpcz_store.cc:269] 0504 13:56:13.626717 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:13.627002 (+ 285us) service_pool.cc:224] Handling call
0504 13:56:14.737585 (+1110583us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:13.725544 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:13.725906 (+ 362us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:13.725929 (+ 23us) write_op.cc:435] Acquired schema lock
0504 13:56:13.725941 (+ 12us) tablet.cc:662] Decoding operations
0504 13:56:13.726243 (+ 302us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:13.726276 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:13.726296 (+ 20us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:13.726525 (+ 229us) tablet.cc:701] Row locks acquired
0504 13:56:13.726537 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:13.726648 (+ 111us) write_op.cc:270] Start()
0504 13:56:13.726721 (+ 73us) write_op.cc:276] Timestamp: P: 1777902973726600 usec, L: 0
0504 13:56:13.726735 (+ 14us) op_driver.cc:348] REPLICATION: starting
0504 13:56:13.727230 (+ 495us) log.cc:844] Serialized 913 byte log entry
0504 13:56:13.913997 (+186767us) op_driver.cc:464] REPLICATION: finished
0504 13:56:13.915009 (+ 1012us) write_op.cc:301] APPLY: starting
0504 13:56:13.915072 (+ 63us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:13.920760 (+ 5688us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:13.920854 (+ 94us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:13.921042 (+ 188us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:13.921172 (+ 130us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:13.921213 (+ 41us) write_op.cc:312] APPLY: finished
0504 13:56:13.921367 (+ 154us) log.cc:844] Serialized 29 byte log entry
0504 13:56:13.921500 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:13.921564 (+ 64us) write_op.cc:454] Released schema lock
0504 13:56:13.921594 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:13.921824 (+ 230us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:13.921851 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":857,"apply.run_cpu_time_us":1219,"apply.run_wall_time_us":6778,"num_ops":2,"prepare.queue_time_us":240,"prepare.run_cpu_time_us":1973,"prepare.run_wall_time_us":3520,"raft.queue_time_us":1334,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":13,"replication_time_us":187170}]]}
I20260504 13:56:14.742198 9664 tablet_service.cc:1558] Processing DeleteTablet for tablet 03818e34ac0545be880c873cf82cd4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:14.745810 9338 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:38691 (request call id 31) took 805 ms (client timeout 1000 ms). Trace:
I20260504 13:56:14.750324 9810 tablet_service.cc:1511] Processing CreateTablet for tablet 3c43ff1d759a40d68344f25f066416ac (DEFAULT_TABLE table=default.Table_1f5a66e30f35482dab7c556daebc87c9 [id=c268fa33d24540b4978e3feebb0f793f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:14.752418 9664 tablet_service.cc:1511] Processing CreateTablet for tablet 07a428049fdb4a68aae729400e2d9674 (DEFAULT_TABLE table=default.Table_0082bd435ced4fdcaeafe8848d05f9bf [id=386315c5331647659a31dc8a2436d7ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:14.753966 9664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a428049fdb4a68aae729400e2d9674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:14.758430 9809 tablet_service.cc:1511] Processing CreateTablet for tablet 07a428049fdb4a68aae729400e2d9674 (DEFAULT_TABLE table=default.Table_0082bd435ced4fdcaeafe8848d05f9bf [id=386315c5331647659a31dc8a2436d7ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:14.762844 9663 tablet_service.cc:1511] Processing CreateTablet for tablet 03fe4ce4f52f48b7b7a804c269e0fd4e (DEFAULT_TABLE table=default.Table_963c93d5d8ed49e6a2052fa1e652da91 [id=399a65bb2fa6428296341de74e964518]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:14.763888 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03fe4ce4f52f48b7b7a804c269e0fd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:14.766885 10013 tablet_bootstrap.cc:492] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:14.767293 10013 ts_tablet_manager.cc:1403] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.041s user 0.018s sys 0.002s
I20260504 13:56:14.769515 10013 raft_consensus.cc:359] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.770084 10013 raft_consensus.cc:385] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:14.770326 10013 raft_consensus.cc:740] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:14.770848 10013 consensus_queue.cc:260] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.772606 10013 ts_tablet_manager.cc:1434] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260504 13:56:14.773341 10013 tablet_bootstrap.cc:492] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:14.774649 9833 raft_consensus.cc:2468] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:14.776948 9474 leader_election.cc:304] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:14.781404 10003 ts_tablet_manager.cc:1929] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:14.781867 10003 log.cc:1199] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/0867df9ea115418193b6d9d3fc251923
I20260504 13:56:14.782666 10003 ts_tablet_manager.cc:1950] T 0867df9ea115418193b6d9d3fc251923 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:14.783555 10149 raft_consensus.cc:515] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.785461 10149 leader_election.cc:290] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:14.786564 9810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c43ff1d759a40d68344f25f066416ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:14.788869 9809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a428049fdb4a68aae729400e2d9674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:14.789480 10013 tablet_bootstrap.cc:654] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:14.792448 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e3b36e1f654a3a8df51480801dabd7" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:14.793154 9685 raft_consensus.cc:3060] T e0e3b36e1f654a3a8df51480801dabd7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.747736 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_221d2c2a7a5c4dc0a6dd2653a4002e7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:14.800144 9338 rpcz_store.cc:269] 0504 13:56:13.940380 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:13.940503 (+ 123us) service_pool.cc:224] Handling call
0504 13:56:14.032554 (+ 92051us) spinlock_profiling.cc:243] Waited 81.5 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:14.745753 (+713199us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":4,"spinlock_wait_cycles":187351808}
W20260504 13:56:14.801290 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_221d2c2a7a5c4dc0a6dd2653a4002e7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:14.785975 10133 raft_consensus.cc:2804] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:14.814541 10133 raft_consensus.cc:697] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:14.817623 10057 raft_consensus.cc:493] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:14.818071 10057 raft_consensus.cc:515] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.820355 10057 leader_election.cc:290] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:14.821568 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdf1c3cef3334a849f7b5e429b7f4990" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:14.822113 9833 raft_consensus.cc:2468] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:14.823127 9620 leader_election.cc:304] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:14.823850 10057 raft_consensus.cc:2804] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:14.824234 10057 raft_consensus.cc:493] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:14.824513 10057 raft_consensus.cc:3060] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:14.825709 10133 consensus_queue.cc:237] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.834826 10013 tablet_bootstrap.cc:492] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:14.835391 10013 ts_tablet_manager.cc:1403] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.062s user 0.012s sys 0.009s
I20260504 13:56:14.838411 10013 raft_consensus.cc:359] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.839145 10013 raft_consensus.cc:385] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:14.839440 10013 raft_consensus.cc:740] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:14.840065 10013 consensus_queue.cc:260] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.851512 9341 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 44017e2079be49258bab49325e3da1c1 (table default.Table_f4a4640b6bb0432dad74376a7ed31181 [id=85b12280c48642b2a9cc8c51af037f2e]). Expected version 1 got 0
I20260504 13:56:14.857218 10013 ts_tablet_manager.cc:1434] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.022s user 0.002s sys 0.004s
I20260504 13:56:14.858286 10013 tablet_bootstrap.cc:492] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:14.859979 9344 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 70caac2e256d419dbc2e70cb5dfffe97 consensus state has no leader
I20260504 13:56:14.865001 10013 tablet_bootstrap.cc:654] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:14.879755 10003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 62) took 1225 ms. Trace:
I20260504 13:56:14.880225 10003 rpcz_store.cc:276] 0504 13:56:13.654168 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:13.654297 (+ 129us) service_pool.cc:224] Handling call
0504 13:56:14.346270 (+691973us) ts_tablet_manager.cc:1151] Deleting tablet 0867df9ea115418193b6d9d3fc251923
0504 13:56:14.346338 (+ 68us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:14.688581 (+342243us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:14.781226 (+ 92645us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:14.879679 (+ 98453us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3,"tablet-delete.queue_time_us":634409}
I20260504 13:56:14.881886 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 0867df9ea115418193b6d9d3fc251923 (table default.Table_39d77929b2fb4ca585977d4c8a362129 [id=92ea89acf1fc427690ee0a61cbbb1647]) successfully deleted
I20260504 13:56:14.882851 10003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 63) took 1224 ms. Trace:
I20260504 13:56:14.883265 10003 rpcz_store.cc:276] 0504 13:56:13.658871 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:13.658972 (+ 101us) service_pool.cc:224] Handling call
0504 13:56:14.882621 (+1223649us) ts_tablet_manager.cc:1151] Deleting tablet 0867df9ea115418193b6d9d3fc251923
0504 13:56:14.882658 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:14.882797 (+ 139us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1170781}
W20260504 13:56:14.884758 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 0867df9ea115418193b6d9d3fc251923 because the tablet was not found. No further retry: Not found: Tablet not found: 0867df9ea115418193b6d9d3fc251923
I20260504 13:56:14.885720 10003 tablet_replica.cc:333] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:14.886843 10003 raft_consensus.cc:2243] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:14.887395 10003 raft_consensus.cc:2272] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:14.889344 10003 ts_tablet_manager.cc:1916] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:14.897544 9685 raft_consensus.cc:2468] T e0e3b36e1f654a3a8df51480801dabd7 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:14.903759 9473 leader_election.cc:304] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:14.904568 10149 raft_consensus.cc:2804] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:14.905081 10149 raft_consensus.cc:697] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:14.906226 10149 consensus_queue.cc:237] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.913468 10013 tablet_bootstrap.cc:492] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:14.914045 10013 ts_tablet_manager.cc:1403] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.007s
I20260504 13:56:14.915752 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdf1c3cef3334a849f7b5e429b7f4990" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:14.916599 9540 raft_consensus.cc:2468] T bdf1c3cef3334a849f7b5e429b7f4990 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:14.921443 10013 raft_consensus.cc:359] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.922255 10013 raft_consensus.cc:385] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:14.926766 10013 raft_consensus.cc:740] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:14.927559 10013 consensus_queue.cc:260] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:14.940591 10013 ts_tablet_manager.cc:1434] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260504 13:56:14.944250 10013 tablet_bootstrap.cc:492] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:14.953611 9472 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.Ping -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 57047us
I20260504 13:56:14.955857 9513 tablet_service.cc:1511] Processing CreateTablet for tablet 3c43ff1d759a40d68344f25f066416ac (DEFAULT_TABLE table=default.Table_1f5a66e30f35482dab7c556daebc87c9 [id=c268fa33d24540b4978e3feebb0f793f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:14.957283 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c43ff1d759a40d68344f25f066416ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:14.967303 10013 tablet_bootstrap.cc:654] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:14.970752 9328 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 3c43ff1d759a40d68344f25f066416ac (table default.Table_1f5a66e30f35482dab7c556daebc87c9 [id=c268fa33d24540b4978e3feebb0f793f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3c43ff1d759a40d68344f25f066416ac consensus state has no leader
I20260504 13:56:14.982314 9660 tablet_service.cc:1511] Processing CreateTablet for tablet 46fdf0ffaf454689a705949bd39deb26 (DEFAULT_TABLE table=default.Table_276161949f364064b8702a62c6beb9e4 [id=e68beb5025184dcb9c24037b6b595293]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:14.983544 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46fdf0ffaf454689a705949bd39deb26. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:14.992061 9911 log.cc:927] Time spent T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
W20260504 13:56:14.996040 9618 consensus_peers.cc:597] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:14.894516 9808 tablet_service.cc:1511] Processing CreateTablet for tablet 03fe4ce4f52f48b7b7a804c269e0fd4e (DEFAULT_TABLE table=default.Table_963c93d5d8ed49e6a2052fa1e652da91 [id=399a65bb2fa6428296341de74e964518]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:15.001816 9808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03fe4ce4f52f48b7b7a804c269e0fd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:15.005405 9517 tablet_service.cc:1558] Processing DeleteTablet for tablet 0b7a71dbcc694e1a97019188cfb2c672 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:15.006445 9517 tablet_service.cc:1558] Processing DeleteTablet for tablet d7e556f81ec940f1aa3c55b3a25f31ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:15.008069 9517 tablet_service.cc:1511] Processing CreateTablet for tablet 07a428049fdb4a68aae729400e2d9674 (DEFAULT_TABLE table=default.Table_0082bd435ced4fdcaeafe8848d05f9bf [id=386315c5331647659a31dc8a2436d7ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:15.009070 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a428049fdb4a68aae729400e2d9674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:15.031996 9342 master_service.cc:954] ReplaceTablet: received request to replace tablet 91371bf848f5465ea2db0c090583382d from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:15.033078 9344 master_service.cc:954] ReplaceTablet: received request to replace tablet d7e556f81ec940f1aa3c55b3a25f31ab from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:15.056325 9344 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_56891f8543a84e859c39b381625a9daa" } modify_external_catalogs: true
I20260504 13:56:15.056809 9344 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_56891f8543a84e859c39b381625a9daa" } modify_external_catalogs: true
I20260504 13:56:15.063647 9813 tablet_service.cc:1511] Processing CreateTablet for tablet 46fdf0ffaf454689a705949bd39deb26 (DEFAULT_TABLE table=default.Table_276161949f364064b8702a62c6beb9e4 [id=e68beb5025184dcb9c24037b6b595293]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:15.069756 10095 consensus_queue.cc:1048] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:15.072271 9515 tablet_service.cc:1511] Processing CreateTablet for tablet 03fe4ce4f52f48b7b7a804c269e0fd4e (DEFAULT_TABLE table=default.Table_963c93d5d8ed49e6a2052fa1e652da91 [id=399a65bb2fa6428296341de74e964518]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:15.073684 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03fe4ce4f52f48b7b7a804c269e0fd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:15.075870 9514 tablet_service.cc:1511] Processing CreateTablet for tablet 46fdf0ffaf454689a705949bd39deb26 (DEFAULT_TABLE table=default.Table_276161949f364064b8702a62c6beb9e4 [id=e68beb5025184dcb9c24037b6b595293]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:15.076894 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46fdf0ffaf454689a705949bd39deb26. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:15.089812 9336 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet afee197774db411d97468913a207703a consensus state has no leader
I20260504 13:56:15.090888 9339 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 07a428049fdb4a68aae729400e2d9674
I20260504 13:56:15.097865 9665 tablet_service.cc:1558] Processing DeleteTablet for tablet 07a428049fdb4a68aae729400e2d9674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:14 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:15.106040 9814 tablet_service.cc:1558] Processing DeleteTablet for tablet 07a428049fdb4a68aae729400e2d9674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:14 UTC) from {username='slave'} at 127.0.0.1:56380
W20260504 13:56:15.107879 10168 log.cc:927] Time spent T b320a37b29844c7593a12000eaa3da43 P 7b79fe7dd8934dd996393a2d96eabf00: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260504 13:56:15.123505 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46fdf0ffaf454689a705949bd39deb26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:15.141921 10162 consensus_queue.cc:1048] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:15.147608 10087 consensus_queue.cc:1048] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:15.154065 9932 tablet_bootstrap.cc:492] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:15.154464 9932 ts_tablet_manager.cc:1403] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.597s user 0.060s sys 0.057s
I20260504 13:56:15.156981 9932 raft_consensus.cc:359] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.157661 9932 raft_consensus.cc:385] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:15.157932 9931 tablet_bootstrap.cc:700] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888: Time spent opening tablet: real 0.519s user 0.004s sys 0.000s
I20260504 13:56:15.157963 9932 raft_consensus.cc:740] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:15.158510 9932 consensus_queue.cc:260] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.160851 9932 ts_tablet_manager.cc:1434] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260504 13:56:15.161854 9932 tablet_bootstrap.cc:492] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:15.167804 10086 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260504 13:56:15.168625 10086 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260504 13:56:15.170342 10086 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260504 13:56:15.173252 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_2aa57c2b774344218cd8fb99ed51879c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:15.174746 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2aa57c2b774344218cd8fb99ed51879c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:15.175225 9931 tablet_bootstrap.cc:654] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:15.182729 9932 tablet_bootstrap.cc:654] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:14.682852 10002 ts_tablet_manager.cc:1916] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:15.214003 9337 catalog_manager.cc:5671] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:15.223372 9665 tablet_service.cc:1511] Processing CreateTablet for tablet 70caac2e256d419dbc2e70cb5dfffe97 (DEFAULT_TABLE table=default.Table_c5584202645f4ef78e4f6e00aa2bf2c1 [id=2665806fb81f4eda8981fede847cbfcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:15.224304 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70caac2e256d419dbc2e70cb5dfffe97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:15.255442 10057 raft_consensus.cc:515] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:15.270334 10086 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260504 13:56:15.270800 10086 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260504 13:56:15.271200 10086 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260504 13:56:15.271706 10086 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260504 13:56:15.277292 10057 leader_election.cc:290] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:15.277499 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdf1c3cef3334a849f7b5e429b7f4990" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:15.278110 9832 raft_consensus.cc:3060] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:15.282997 9814 tablet_service.cc:1511] Processing CreateTablet for tablet 70caac2e256d419dbc2e70cb5dfffe97 (DEFAULT_TABLE table=default.Table_c5584202645f4ef78e4f6e00aa2bf2c1 [id=2665806fb81f4eda8981fede847cbfcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:15.284013 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70caac2e256d419dbc2e70cb5dfffe97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:15.303025 9665 tablet_service.cc:1511] Processing CreateTablet for tablet c34877a81d504f46a2f680c5d5ddb4f4 (DEFAULT_TABLE table=default.Table_39d77929b2fb4ca585977d4c8a362129 [id=92ea89acf1fc427690ee0a61cbbb1647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:15.304018 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34877a81d504f46a2f680c5d5ddb4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:15.319002 10057 raft_consensus.cc:493] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:15.319677 10057 raft_consensus.cc:515] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.321290 9932 tablet_bootstrap.cc:492] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:15.321900 9932 ts_tablet_manager.cc:1403] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.160s user 0.028s sys 0.022s
I20260504 13:56:15.322636 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e556f81ec940f1aa3c55b3a25f31ab" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:15.323196 9931 tablet_bootstrap.cc:492] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:15.323568 9931 ts_tablet_manager.cc:1403] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.689s user 0.010s sys 0.000s
I20260504 13:56:15.324203 9833 raft_consensus.cc:2393] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5beceb6317444247b31b44a6581707c6 in current term 1: Already voted for candidate d4051d2bf0bf410eb8acd107560df888 in this term.
I20260504 13:56:15.324316 9932 raft_consensus.cc:359] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.325044 9932 raft_consensus.cc:385] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:15.325261 9932 raft_consensus.cc:740] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:15.326252 10057 leader_election.cc:290] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:15.240001 9336 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_737b85d6e3e04656a60981e69282cd7b" } new_table_name: "default.Table_960fc67a53724e78bcb430e7fdef3151" modify_external_catalogs: true
I20260504 13:56:15.329705 9932 consensus_queue.cc:260] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.240887 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_51968c36cc6e4b65a5135e9b2cde5296"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:15.332376 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51968c36cc6e4b65a5135e9b2cde5296 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:15.251637 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_f7b06aadc0024855b08b56cad8c7b6a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:15.336122 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7b06aadc0024855b08b56cad8c7b6a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:15.294005 9340 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_ec713dc211a9418f9f9773e25ea10095" } new_table_name: "default.Table_d0881bfb141241bd8a58a97a29384b23" modify_external_catalogs: true
I20260504 13:56:15.298795 9335 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_f4a4640b6bb0432dad74376a7ed31181" } modify_external_catalogs: true
I20260504 13:56:15.339394 9335 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_f4a4640b6bb0432dad74376a7ed31181" } modify_external_catalogs: true
I20260504 13:56:15.331596 9932 ts_tablet_manager.cc:1434] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260504 13:56:15.342722 9932 tablet_bootstrap.cc:492] T b37a738f2902402ca76e4edebcdba474 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:15.345758 9344 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 5625163f6add4a20b1828905d90b5486
I20260504 13:56:15.345490 9931 raft_consensus.cc:359] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.346125 9931 raft_consensus.cc:385] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:15.346362 9931 raft_consensus.cc:740] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:15.352855 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet e0e3b36e1f654a3a8df51480801dabd7 (table default.Table_9e9ef3353dcf4b1fadaa2f51bc8f7041 [id=8dcabd7d67334598926f927a62e630d6]). Expected version 2 got 0
I20260504 13:56:15.353485 9832 raft_consensus.cc:2468] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:15.354810 9620 leader_election.cc:304] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:15.355643 10057 raft_consensus.cc:2804] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:15.356052 10057 raft_consensus.cc:697] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:15.357245 9660 tablet_service.cc:1558] Processing DeleteTablet for tablet 5625163f6add4a20b1828905d90b5486 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:15 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:15.357968 9944 ts_tablet_manager.cc:1929] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:15.357577 10057 consensus_queue.cc:237] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.358301 9944 log.cc:1199] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/b4dca0cadd5e400eab1365c03a8de4f4
I20260504 13:56:15.358955 9944 ts_tablet_manager.cc:1950] T b4dca0cadd5e400eab1365c03a8de4f4 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:15.359649 9931 consensus_queue.cc:260] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.360476 9812 tablet_service.cc:1558] Processing DeleteTablet for tablet 5625163f6add4a20b1828905d90b5486 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:15 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:15.363466 9932 tablet_bootstrap.cc:654] T b37a738f2902402ca76e4edebcdba474 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:15.370005 9931 ts_tablet_manager.cc:1434] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.046s user 0.006s sys 0.000s
I20260504 13:56:15.371002 9343 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: default.Table_f4a4640b6bb0432dad74376a7ed31181 [id=85b12280c48642b2a9cc8c51af037f2e] alter complete (version 1)
I20260504 13:56:15.373324 9931 tablet_bootstrap.cc:492] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:15.376766 9663 tablet_service.cc:1511] Processing CreateTablet for tablet b8e4239942f34fb081a026a35740e465 (DEFAULT_TABLE table=default.Table_895e961d6c1a45e688ad4e0ca752bae9 [id=73e12524eb9e416782bf9f7fb58c93bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:15.377810 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e4239942f34fb081a026a35740e465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:15.378322 9931 tablet_bootstrap.cc:654] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:15.379102 9813 tablet_service.cc:1511] Processing CreateTablet for tablet c34877a81d504f46a2f680c5d5ddb4f4 (DEFAULT_TABLE table=default.Table_39d77929b2fb4ca585977d4c8a362129 [id=92ea89acf1fc427690ee0a61cbbb1647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:15.379781 9932 tablet_bootstrap.cc:492] T b37a738f2902402ca76e4edebcdba474 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:15.380142 9932 ts_tablet_manager.cc:1403] T b37a738f2902402ca76e4edebcdba474 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.000s
I20260504 13:56:15.382308 9932 raft_consensus.cc:359] T b37a738f2902402ca76e4edebcdba474 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.383046 9932 raft_consensus.cc:385] T b37a738f2902402ca76e4edebcdba474 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:15.383311 9932 raft_consensus.cc:740] T b37a738f2902402ca76e4edebcdba474 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:15.387411 9932 consensus_queue.cc:260] T b37a738f2902402ca76e4edebcdba474 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.396718 9932 ts_tablet_manager.cc:1434] T b37a738f2902402ca76e4edebcdba474 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260504 13:56:15.399670 9812 tablet_service.cc:1511] Processing CreateTablet for tablet b8e4239942f34fb081a026a35740e465 (DEFAULT_TABLE table=default.Table_895e961d6c1a45e688ad4e0ca752bae9 [id=73e12524eb9e416782bf9f7fb58c93bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:15.401242 9932 tablet_bootstrap.cc:492] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:15.405218 9660 tablet_service.cc:1511] Processing CreateTablet for tablet 1501455a0af348e38746d100987afdd0 (DEFAULT_TABLE table=default.Table_9b8d10cfdddc499a86ba289a87dbe689 [id=1fcd0497578846cb9bbd71e501606912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:15.406260 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1501455a0af348e38746d100987afdd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:15.416771 9932 tablet_bootstrap.cc:654] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:15.421614 9809 tablet_service.cc:1511] Processing CreateTablet for tablet 1501455a0af348e38746d100987afdd0 (DEFAULT_TABLE table=default.Table_9b8d10cfdddc499a86ba289a87dbe689 [id=1fcd0497578846cb9bbd71e501606912]), partition=RANGE (key) PARTITION UNBOUNDED
W20260504 13:56:15.442508 9336 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a1cf52a9e58744a99dae1958d2c1f867 has no consensus state
I20260504 13:56:15.445475 9932 tablet_bootstrap.cc:492] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:15.445842 9932 ts_tablet_manager.cc:1403] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.008s
I20260504 13:56:15.448618 9932 raft_consensus.cc:359] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.449345 9932 raft_consensus.cc:385] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:15.449642 9932 raft_consensus.cc:740] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:15.449992 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 64) took 1301 ms. Trace:
I20260504 13:56:15.450340 9944 rpcz_store.cc:276] 0504 13:56:14.148848 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:14.151280 (+ 2432us) service_pool.cc:224] Handling call
0504 13:56:14.386891 (+235611us) ts_tablet_manager.cc:1151] Deleting tablet b4dca0cadd5e400eab1365c03a8de4f4
0504 13:56:14.386934 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:15.140416 (+753482us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:15.357860 (+217444us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:15.449934 (+ 92074us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":235089}
I20260504 13:56:15.452468 9812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e4239942f34fb081a026a35740e465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:15.452425 10003 ts_tablet_manager.cc:1929] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:15.452872 10003 log.cc:1199] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/67f26d71286c4f90abd681f9fc9d1e30
I20260504 13:56:15.453526 10003 ts_tablet_manager.cc:1950] T 67f26d71286c4f90abd681f9fc9d1e30 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:15.451985 9944 tablet_replica.cc:333] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:15.450395 9932 consensus_queue.cc:260] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.454836 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet b4dca0cadd5e400eab1365c03a8de4f4 (table default.Table_15bf9089e68d4882838fe4f1c65156ef [id=3814e135130a4213a46df3cd93eca49e]) successfully deleted
I20260504 13:56:15.455896 9809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1501455a0af348e38746d100987afdd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:15.456776 10136 tablet.cc:1724] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260504 13:56:15.457177 9944 raft_consensus.cc:2243] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:15.457623 9944 raft_consensus.cc:2272] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:15.459022 9932 ts_tablet_manager.cc:1434] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260504 13:56:15.460668 9944 ts_tablet_manager.cc:1916] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:15.461233 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34877a81d504f46a2f680c5d5ddb4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:15.465145 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_a660204a2583425aa3ecd09c5cc01368"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:15.466842 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a660204a2583425aa3ecd09c5cc01368 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:15.474987 10003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 66) took 1710 ms. Trace:
I20260504 13:56:15.475478 10003 rpcz_store.cc:276] 0504 13:56:13.764679 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:13.764826 (+ 147us) service_pool.cc:224] Handling call
0504 13:56:14.885589 (+1120763us) ts_tablet_manager.cc:1151] Deleting tablet 67f26d71286c4f90abd681f9fc9d1e30
0504 13:56:14.885624 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:15.349570 (+463946us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:15.452290 (+102720us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:15.474914 (+ 22624us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1120270}
I20260504 13:56:15.477027 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 67f26d71286c4f90abd681f9fc9d1e30 (table default.Table_895e961d6c1a45e688ad4e0ca752bae9 [id=73e12524eb9e416782bf9f7fb58c93bd]) successfully deleted
I20260504 13:56:15.478147 10003 tablet_replica.cc:333] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:15.479100 10003 raft_consensus.cc:2243] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:15.479727 10003 raft_consensus.cc:2272] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:15.481730 10003 ts_tablet_manager.cc:1916] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:15.506498 9932 tablet_bootstrap.cc:492] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:15.525853 9931 tablet_bootstrap.cc:492] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:15.533484 9931 ts_tablet_manager.cc:1403] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.162s user 0.017s sys 0.004s
I20260504 13:56:15.541275 9932 tablet_bootstrap.cc:654] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:15.543741 9931 raft_consensus.cc:359] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.553978 9931 raft_consensus.cc:385] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:15.554843 9931 raft_consensus.cc:740] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:15.556170 9931 consensus_queue.cc:260] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.564386 9336 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_221d2c2a7a5c4dc0a6dd2653a4002e7f" } modify_external_catalogs: true
W20260504 13:56:15.564725 9620 consensus_peers.cc:597] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0867df9ea115418193b6d9d3fc251923. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:15.565032 9336 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_221d2c2a7a5c4dc0a6dd2653a4002e7f" } modify_external_catalogs: true
I20260504 13:56:15.570999 9931 ts_tablet_manager.cc:1434] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.031s user 0.005s sys 0.001s
I20260504 13:56:15.571866 9931 tablet_bootstrap.cc:492] T b37a738f2902402ca76e4edebcdba474 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:15.582095 9932 tablet_bootstrap.cc:492] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:15.582486 9932 ts_tablet_manager.cc:1403] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.002s
I20260504 13:56:15.585173 9932 raft_consensus.cc:359] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.585726 9932 raft_consensus.cc:385] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:15.585944 9932 raft_consensus.cc:740] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:15.591595 9931 tablet_bootstrap.cc:654] T b37a738f2902402ca76e4edebcdba474 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:15.592026 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_7f9a1296bc424f7aa21625c670da0dd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:15.593600 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f9a1296bc424f7aa21625c670da0dd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:15.502753 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdf1c3cef3334a849f7b5e429b7f4990" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
W20260504 13:56:15.308801 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 531947us
W20260504 13:56:15.492748 10013 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.011s sys 0.042s
W20260504 13:56:15.596139 10013 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.620s user 0.011s sys 0.050s
I20260504 13:56:15.596891 9932 consensus_queue.cc:260] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.597334 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e3b36e1f654a3a8df51480801dabd7" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:15.597810 10013 tablet_bootstrap.cc:492] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:15.597868 9832 raft_consensus.cc:3060] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:15.598142 10013 ts_tablet_manager.cc:1403] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.654s user 0.016s sys 0.056s
I20260504 13:56:15.599496 9932 ts_tablet_manager.cc:1434] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260504 13:56:15.600322 9932 tablet_bootstrap.cc:492] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:15.604504 9537 raft_consensus.cc:3060] T bdf1c3cef3334a849f7b5e429b7f4990 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:15.610888 10013 raft_consensus.cc:359] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.611670 10013 raft_consensus.cc:385] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:15.611958 10013 raft_consensus.cc:740] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:15.612636 10013 consensus_queue.cc:260] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:15.580503 9513 consensus_meta.cc:287] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00: Time spent flushing consensus metadata: real 0.614s user 0.008s sys 0.005s
I20260504 13:56:15.581523 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7e556f81ec940f1aa3c55b3a25f31ab" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:15.620920 9536 raft_consensus.cc:2393] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5beceb6317444247b31b44a6581707c6 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:15.623200 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_f8a23a8252f84cacb13b338f9d9dc9fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:15.624644 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8a23a8252f84cacb13b338f9d9dc9fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:15.632001 9344 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_6c0e4aec8a8a4b35a9c5509f3be15772" } new_table_name: "default.Table_b32f665dddb043769e07076397f84a73" modify_external_catalogs: true
I20260504 13:56:15.649072 9932 tablet_bootstrap.cc:654] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:15.652031 10003 ts_tablet_manager.cc:1929] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:15.652419 10003 log.cc:1199] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/d7e556f81ec940f1aa3c55b3a25f31ab
I20260504 13:56:15.653039 10003 ts_tablet_manager.cc:1950] T d7e556f81ec940f1aa3c55b3a25f31ab P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:15.657985 9832 raft_consensus.cc:2468] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:15.661468 10003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 71) took 1713 ms. Trace:
I20260504 13:56:15.661917 10003 rpcz_store.cc:276] 0504 13:56:13.948306 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:13.948456 (+ 150us) service_pool.cc:224] Handling call
0504 13:56:15.477961 (+1529505us) ts_tablet_manager.cc:1151] Deleting tablet d7e556f81ec940f1aa3c55b3a25f31ab
0504 13:56:15.478010 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:15.547579 (+ 69569us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:15.651906 (+104327us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:15.661399 (+ 9493us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1528900}
I20260504 13:56:15.662534 9618 leader_election.cc:304] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:15.663077 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet d7e556f81ec940f1aa3c55b3a25f31ab (table default.Table_ea18464634544273a0bbb10fce8b88ba [id=2388ca57302443e48f66ac3a8db39000]) successfully deleted
I20260504 13:56:15.663512 10095 raft_consensus.cc:3060] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:15.663424 10086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.238.212:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.238.254:36145,127.8.238.253:36337,127.8.238.252:44781
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.253:36337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.238.253
--webserver_port=44077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 04 May 2026 13:43:23 UTC on e7f111948823
build id 11743
TSAN enabled
I20260504 13:56:15.664491 10003 tablet_replica.cc:333] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:15.665293 10003 raft_consensus.cc:2243] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:15.665793 10003 raft_consensus.cc:2272] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:15.668308 10003 ts_tablet_manager.cc:1916] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:15.669217 10086 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 13:56:15.674829 10057 consensus_queue.cc:1048] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:15.691557 10086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260504 13:56:15.693231 10162 raft_consensus.cc:493] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:15.694922 9932 tablet_bootstrap.cc:492] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:15.695338 9932 ts_tablet_manager.cc:1403] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.004s
I20260504 13:56:15.696954 10162 raft_consensus.cc:515] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.697603 9932 raft_consensus.cc:359] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.698210 9932 raft_consensus.cc:385] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:15.698462 9932 raft_consensus.cc:740] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:15.699154 10162 leader_election.cc:290] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:15.695559 10133 raft_consensus.cc:493] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:15.699319 9932 consensus_queue.cc:260] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.699661 10133 raft_consensus.cc:515] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.701301 10133 leader_election.cc:290] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:15.711651 9932 ts_tablet_manager.cc:1434] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
W20260504 13:56:15.713935 9539 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.001s sys 0.012s
W20260504 13:56:15.714285 9539 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.559s user 0.002s sys 0.012s
I20260504 13:56:15.730911 9932 tablet_bootstrap.cc:492] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:15.739037 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 279341us
I20260504 13:56:15.744838 9932 tablet_bootstrap.cc:654] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:15.759310 10095 raft_consensus.cc:2749] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:15.766211 10003 ts_tablet_manager.cc:1929] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:15.766564 10003 log.cc:1199] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/14b7f1e834d0439c89abb180bd13853d
I20260504 13:56:15.767184 10003 ts_tablet_manager.cc:1950] T 14b7f1e834d0439c89abb180bd13853d P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:15.773262 10095 raft_consensus.cc:493] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:15.773573 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fdc09a498be470ab04563c62caf7755" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:15.773666 10095 raft_consensus.cc:515] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.774155 9684 raft_consensus.cc:2468] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:15.775424 10095 leader_election.cc:290] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:15.785236 9932 tablet_bootstrap.cc:492] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:15.785619 9932 ts_tablet_manager.cc:1403] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.002s
I20260504 13:56:15.788740 9932 raft_consensus.cc:359] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.789341 9932 raft_consensus.cc:385] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:15.789588 9932 raft_consensus.cc:740] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:15.790145 9932 consensus_queue.cc:260] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.796662 10196 raft_consensus.cc:493] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:15.797737 10196 raft_consensus.cc:515] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.799698 10196 leader_election.cc:290] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:15.804963 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14b7f1e834d0439c89abb180bd13853d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:15.806686 9513 tablet_service.cc:1558] Processing DeleteTablet for tablet 07a428049fdb4a68aae729400e2d9674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:14 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:15.809695 9513 tablet_service.cc:1511] Processing CreateTablet for tablet 70caac2e256d419dbc2e70cb5dfffe97 (DEFAULT_TABLE table=default.Table_c5584202645f4ef78e4f6e00aa2bf2c1 [id=2665806fb81f4eda8981fede847cbfcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:15.810740 9932 ts_tablet_manager.cc:1434] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.025s user 0.004s sys 0.000s
I20260504 13:56:15.810846 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70caac2e256d419dbc2e70cb5dfffe97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:15.822871 9932 tablet_bootstrap.cc:492] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:15.826666 10188 tablet.cc:1724] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260504 13:56:15.837720 10003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 72) took 1830 ms. Trace:
I20260504 13:56:15.838079 10003 rpcz_store.cc:276] 0504 13:56:14.007259 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:14.010239 (+ 2980us) service_pool.cc:224] Handling call
0504 13:56:15.664233 (+1653994us) ts_tablet_manager.cc:1151] Deleting tablet 14b7f1e834d0439c89abb180bd13853d
0504 13:56:15.664273 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:15.759222 (+ 94949us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:15.766098 (+ 6876us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:15.837667 (+ 71569us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1653456}
I20260504 13:56:15.839677 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 14b7f1e834d0439c89abb180bd13853d (table default.Table_a0a1002687744b879e47ae4fd701d903 [id=2d45d703594b416893908650027e94b3]) successfully deleted
I20260504 13:56:15.840508 10003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 73) took 1833 ms. Trace:
I20260504 13:56:15.840821 10003 rpcz_store.cc:276] 0504 13:56:14.007593 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:14.055481 (+ 47888us) service_pool.cc:224] Handling call
0504 13:56:15.838968 (+1783487us) ts_tablet_manager.cc:1151] Deleting tablet cdb86f6370bf4c5f9c488d85efb67ef7
0504 13:56:15.839007 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:15.840457 (+ 1450us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1779612}
I20260504 13:56:15.841809 10003 tablet_replica.cc:333] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:15.842504 10003 raft_consensus.cc:2243] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:15.842612 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b37a738f2902402ca76e4edebcdba474" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:15.843185 9684 raft_consensus.cc:2468] T b37a738f2902402ca76e4edebcdba474 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
W20260504 13:56:15.843721 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet cdb86f6370bf4c5f9c488d85efb67ef7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cdb86f6370bf4c5f9c488d85efb67ef7 already in progress: creating tablet
I20260504 13:56:15.842986 10003 raft_consensus.cc:2272] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:15.845340 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ed4f5ea230a413a8dcdbb661ce173b6" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:15.846335 9684 raft_consensus.cc:2468] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:15.846776 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14b7f1e834d0439c89abb180bd13853d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:15.847517 9473 leader_election.cc:304] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
W20260504 13:56:15.847963 9620 leader_election.cc:343] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: Tablet not RUNNING: SHUTDOWN
I20260504 13:56:15.848649 9473 leader_election.cc:304] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:15.849542 9473 leader_election.cc:304] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:15.850672 10194 raft_consensus.cc:2804] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:15.850582 9618 leader_election.cc:304] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:15.851128 10194 raft_consensus.cc:493] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:15.851509 10194 raft_consensus.cc:3060] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:15.851722 10095 raft_consensus.cc:2749] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:15.856456 10003 ts_tablet_manager.cc:1916] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:15.862430 9932 tablet_bootstrap.cc:654] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:15.881184 10206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:56:15.882627 9932 tablet_bootstrap.cc:492] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:15.883342 9932 ts_tablet_manager.cc:1403] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.000s
I20260504 13:56:15.885748 9932 raft_consensus.cc:359] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.886516 9932 raft_consensus.cc:385] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:15.886775 9932 raft_consensus.cc:740] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:15.895640 9932 consensus_queue.cc:260] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.898042 9931 tablet_bootstrap.cc:492] T b37a738f2902402ca76e4edebcdba474 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:15.898478 9931 ts_tablet_manager.cc:1403] T b37a738f2902402ca76e4edebcdba474 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.327s user 0.030s sys 0.037s
I20260504 13:56:15.904469 9932 ts_tablet_manager.cc:1434] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.021s user 0.004s sys 0.001s
I20260504 13:56:15.906400 9932 tablet_bootstrap.cc:492] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:15.915589 10002 ts_tablet_manager.cc:1929] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:15.916028 10002 log.cc:1199] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/03818e34ac0545be880c873cf82cd4ea
I20260504 13:56:15.916779 10002 ts_tablet_manager.cc:1950] T 03818e34ac0545be880c873cf82cd4ea P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:15.918545 9517 tablet_service.cc:1511] Processing CreateTablet for tablet c34877a81d504f46a2f680c5d5ddb4f4 (DEFAULT_TABLE table=default.Table_39d77929b2fb4ca585977d4c8a362129 [id=92ea89acf1fc427690ee0a61cbbb1647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:15.919725 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34877a81d504f46a2f680c5d5ddb4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:15.923435 10200 raft_consensus.cc:2804] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:15.923872 10200 raft_consensus.cc:493] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:15.924249 10200 raft_consensus.cc:3060] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:15.925437 10095 raft_consensus.cc:493] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:15.925822 10095 raft_consensus.cc:515] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.927625 10095 leader_election.cc:290] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:15.934063 9336 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 72b659b78ec442f1bca3fbcbad5bcdca
I20260504 13:56:15.940356 10197 raft_consensus.cc:2804] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:15.940733 10197 raft_consensus.cc:493] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:15.941045 10197 raft_consensus.cc:3060] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:15.947515 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de639fda874248f0853e74ee60aaf758" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:15.949421 9620 leader_election.cc:343] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:15.950904 9339 catalog_manager.cc:5671] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:15.952277 9339 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet e0e3b36e1f654a3a8df51480801dabd7 (table default.Table_9e9ef3353dcf4b1fadaa2f51bc8f7041 [id=8dcabd7d67334598926f927a62e630d6]). Expected version 2 got 0
I20260504 13:56:15.953357 9339 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 39f080a0296b44988e8e51e68453a551 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]). Expected version 4 got 3
I20260504 13:56:15.957451 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de639fda874248f0853e74ee60aaf758" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:15.958068 9540 raft_consensus.cc:2468] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:15.965730 10095 raft_consensus.cc:493] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:15.966167 10095 raft_consensus.cc:515] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:15.967926 10003 ts_tablet_manager.cc:1929] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:15.968276 10003 log.cc:1199] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/b4dca0cadd5e400eab1365c03a8de4f4
I20260504 13:56:15.968924 10003 ts_tablet_manager.cc:1950] T b4dca0cadd5e400eab1365c03a8de4f4 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:15.969439 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a43cdb767fd14ea09b4db4d2881b69aa" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:15.970798 9620 leader_election.cc:343] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:15.971889 10095 leader_election.cc:290] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:15.976373 10003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 74) took 1862 ms. Trace:
I20260504 13:56:15.976722 10003 rpcz_store.cc:276] 0504 13:56:14.114598 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:14.115092 (+ 494us) service_pool.cc:224] Handling call
0504 13:56:15.841680 (+1726588us) ts_tablet_manager.cc:1151] Deleting tablet b4dca0cadd5e400eab1365c03a8de4f4
0504 13:56:15.841715 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:15.855865 (+ 14150us) spinlock_profiling.cc:243] Waited 8.73 ms on lock 0x7b2000020930. stack: 00007fa12409cc83 00007fa12409cb80 00007fa123c0d574 000000000050e094 000000000050e029 00007fa12c1221a1 00007fa12c153633 00007fa12c15b0a2 00007fa12c15b021 00007fa12c15afb1 00007fa12c15af71 00007fa12c159cdd 00000000005b3bc4 00000000005b39f9 00007fa12401d3af 00007fa12b508ea6
0504 13:56:15.868665 (+ 12800us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:15.967802 (+ 99137us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:15.976312 (+ 8510us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20084096,"tablet-delete.queue_time_us":1725555}
I20260504 13:56:15.978684 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet b4dca0cadd5e400eab1365c03a8de4f4 (table default.Table_15bf9089e68d4882838fe4f1c65156ef [id=3814e135130a4213a46df3cd93eca49e]) successfully deleted
I20260504 13:56:15.979671 10003 tablet_replica.cc:333] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:15.980463 10003 raft_consensus.cc:2243] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:15.980950 10003 raft_consensus.cc:2272] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:15.982869 10003 ts_tablet_manager.cc:1916] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:15.981594 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a43cdb767fd14ea09b4db4d2881b69aa" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:15.996743 9932 tablet_bootstrap.cc:654] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:15.999652 9618 leader_election.cc:304] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:16.001411 10095 raft_consensus.cc:2804] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:16.001729 10095 raft_consensus.cc:493] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:16.002014 10095 raft_consensus.cc:3060] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:16.007584 9618 leader_election.cc:343] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:16.007992 9618 leader_election.cc:304] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:16.009919 10057 raft_consensus.cc:2749] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:16.015852 9330 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 70caac2e256d419dbc2e70cb5dfffe97 (table default.Table_c5584202645f4ef78e4f6e00aa2bf2c1 [id=2665806fb81f4eda8981fede847cbfcf], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 70caac2e256d419dbc2e70cb5dfffe97 consensus state has no leader
I20260504 13:56:16.015880 9515 tablet_service.cc:1558] Processing DeleteTablet for tablet 5625163f6add4a20b1828905d90b5486 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:15 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:16.017750 9515 tablet_service.cc:1511] Processing CreateTablet for tablet b8e4239942f34fb081a026a35740e465 (DEFAULT_TABLE table=default.Table_895e961d6c1a45e688ad4e0ca752bae9 [id=73e12524eb9e416782bf9f7fb58c93bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:16.019162 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e4239942f34fb081a026a35740e465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:16.024268 9932 tablet_bootstrap.cc:492] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:16.024665 9932 ts_tablet_manager.cc:1403] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.118s user 0.009s sys 0.000s
I20260504 13:56:16.029872 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fdc09a498be470ab04563c62caf7755" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:16.030493 9833 raft_consensus.cc:2468] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:16.032403 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ed4f5ea230a413a8dcdbb661ce173b6" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:16.033634 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b37a738f2902402ca76e4edebcdba474" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:16.046968 9340 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_39d77929b2fb4ca585977d4c8a362129" } new_table_name: "default.Table_c6a898ba23b74b54bb4684e970b44742" modify_external_catalogs: true
I20260504 13:56:16.026940 9932 raft_consensus.cc:359] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.049578 9932 raft_consensus.cc:385] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:16.049829 9932 raft_consensus.cc:740] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:16.050437 9932 consensus_queue.cc:260] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.058120 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_597ab174935c4601914d71c16e0c2620"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:16.059983 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_597ab174935c4601914d71c16e0c2620 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:16.033509 9514 tablet_service.cc:1511] Processing CreateTablet for tablet 1501455a0af348e38746d100987afdd0 (DEFAULT_TABLE table=default.Table_9b8d10cfdddc499a86ba289a87dbe689 [id=1fcd0497578846cb9bbd71e501606912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:16.077037 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1501455a0af348e38746d100987afdd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:15.934387 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 67) took 1981 ms. Trace:
W20260504 13:56:15.933764 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 84295.1us
I20260504 13:56:16.080595 10197 raft_consensus.cc:515] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.085263 10197 leader_election.cc:290] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:16.080744 10215 raft_consensus.cc:493] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:16.088552 10215 raft_consensus.cc:515] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.087697 9932 ts_tablet_manager.cc:1434] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.063s user 0.005s sys 0.002s
I20260504 13:56:16.094195 9932 tablet_bootstrap.cc:492] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:16.108570 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3793fe8f1ec43ad809403c4593f5ef6" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:16.109161 9833 raft_consensus.cc:2468] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:16.110275 9620 leader_election.cc:304] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:16.111294 10057 raft_consensus.cc:2804] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:16.111622 10057 raft_consensus.cc:493] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:16.111922 10057 raft_consensus.cc:3060] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:16.112859 9344 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b13303d2a14c4187b82da9f25ce9753e has no consensus state
I20260504 13:56:16.118440 10215 leader_election.cc:290] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:16.120625 9931 raft_consensus.cc:359] T b37a738f2902402ca76e4edebcdba474 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.123283 9931 raft_consensus.cc:385] T b37a738f2902402ca76e4edebcdba474 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:16.129875 9932 tablet_bootstrap.cc:654] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:16.132205 9931 raft_consensus.cc:740] T b37a738f2902402ca76e4edebcdba474 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:16.137786 9344 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_51968c36cc6e4b65a5135e9b2cde5296" } modify_external_catalogs: true
I20260504 13:56:16.138257 9344 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_51968c36cc6e4b65a5135e9b2cde5296" } modify_external_catalogs: true
I20260504 13:56:16.134958 9931 consensus_queue.cc:260] T b37a738f2902402ca76e4edebcdba474 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.146679 10013 ts_tablet_manager.cc:1434] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.548s user 0.114s sys 0.117s
W20260504 13:56:16.147109 10013 ts_tablet_manager.cc:1467] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00: Tablet startup took 548ms
I20260504 13:56:16.153945 9684 raft_consensus.cc:3060] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:16.152601 10013 ts_tablet_manager.cc:1469] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00: Trace:
0504 13:56:12.699122 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:12.721263 (+ 22141us) service_pool.cc:224] Handling call
0504 13:56:12.722542 (+ 1279us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:12.722637 (+ 95us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:12.932052 (+209415us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:12.968941 (+ 36889us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:12.969325 (+ 384us) inbound_call.cc:177] Queueing success response
0504 13:56:14.943766 (+1974441us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0504 13:56:15.598373 (+654607us) ts_tablet_manager.cc:1435] Starting tablet replica
0504 13:56:15.598856 (+ 483us) tablet_replica.cc:261] Starting instrumentation
0504 13:56:15.599320 (+ 464us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":10147,"raft.queue_time_us":81574,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":169,"tablet-open.queue_time_us":1972057,"thread_start_us":471954,"threads_started":1}
W20260504 13:56:16.166357 9537 consensus_meta.cc:287] T bdf1c3cef3334a849f7b5e429b7f4990 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent flushing consensus metadata: real 0.561s user 0.008s sys 0.000s
I20260504 13:56:16.166666 9537 raft_consensus.cc:2468] T bdf1c3cef3334a849f7b5e429b7f4990 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:16.167546 10057 raft_consensus.cc:515] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.170214 10057 leader_election.cc:290] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:16.174073 9932 tablet_bootstrap.cc:492] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:16.174474 9932 ts_tablet_manager.cc:1403] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.083s user 0.010s sys 0.001s
I20260504 13:56:16.178957 9944 ts_tablet_manager.cc:1929] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:16.179472 9944 log.cc:1199] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/0b7a71dbcc694e1a97019188cfb2c672
I20260504 13:56:16.180260 9944 ts_tablet_manager.cc:1950] T 0b7a71dbcc694e1a97019188cfb2c672 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:16.184032 9932 raft_consensus.cc:359] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.184643 9932 raft_consensus.cc:385] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:16.184878 9932 raft_consensus.cc:740] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:16.186710 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3793fe8f1ec43ad809403c4593f5ef6" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:16.197140 9342 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 91371bf848f5465ea2db0c090583382d
I20260504 13:56:16.198479 9342 catalog_manager.cc:6741] ReplaceTablet: tablet 91371bf848f5465ea2db0c090583382d deleted and replaced by tablet 60d3cc59ffe947fbb57b7d3301ccbac1
I20260504 13:56:16.198797 9342 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:35768 (request call id 130) took 1167 ms. Trace:
I20260504 13:56:16.199041 10013 tablet_bootstrap.cc:492] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:16.202732 9932 consensus_queue.cc:260] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.199091 9342 rpcz_store.cc:276] 0504 13:56:15.031737 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:15.031879 (+ 142us) service_pool.cc:224] Handling call
0504 13:56:16.198745 (+1166866us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:16.182249 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:16.182759 (+ 510us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:16.182789 (+ 30us) write_op.cc:435] Acquired schema lock
0504 13:56:16.182808 (+ 19us) tablet.cc:662] Decoding operations
0504 13:56:16.183230 (+ 422us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:16.183273 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:16.183292 (+ 19us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:16.183464 (+ 172us) tablet.cc:701] Row locks acquired
0504 13:56:16.183476 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:16.183571 (+ 95us) write_op.cc:270] Start()
0504 13:56:16.183642 (+ 71us) write_op.cc:276] Timestamp: P: 1777902976183531 usec, L: 0
0504 13:56:16.183656 (+ 14us) op_driver.cc:348] REPLICATION: starting
0504 13:56:16.184072 (+ 416us) log.cc:844] Serialized 662 byte log entry
0504 13:56:16.194563 (+ 10491us) op_driver.cc:464] REPLICATION: finished
0504 13:56:16.194853 (+ 290us) write_op.cc:301] APPLY: starting
0504 13:56:16.194936 (+ 83us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:16.195036 (+ 100us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:16.195188 (+ 152us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:16.195601 (+ 413us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:16.195814 (+ 213us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:16.195875 (+ 61us) write_op.cc:312] APPLY: finished
0504 13:56:16.196051 (+ 176us) log.cc:844] Serialized 29 byte log entry
0504 13:56:16.196199 (+ 148us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:16.196271 (+ 72us) write_op.cc:454] Released schema lock
0504 13:56:16.196311 (+ 40us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":169,"apply.run_cpu_time_us":1643,"apply.run_wall_time_us":1643,"num_ops":2,"prepare.queue_time_us":231450,"raft.queue_time_us":266,"raft.run_cpu_time_us":502,"raft.run_wall_time_us":503,"replication_time_us":10833}]]}
I20260504 13:56:16.211833 10095 raft_consensus.cc:515] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.220613 10095 leader_election.cc:290] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:16.223798 10003 ts_tablet_manager.cc:1929] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:16.224217 10003 log.cc:1199] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/0b7a71dbcc694e1a97019188cfb2c672
I20260504 13:56:16.224889 10003 ts_tablet_manager.cc:1950] T 0b7a71dbcc694e1a97019188cfb2c672 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:16.081773 10002 rpcz_store.cc:276] 0504 13:56:13.936123 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:13.936583 (+ 460us) service_pool.cc:224] Handling call
0504 13:56:14.680067 (+743484us) ts_tablet_manager.cc:1151] Deleting tablet 03818e34ac0545be880c873cf82cd4ea
0504 13:56:14.680102 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:15.350794 (+670692us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:15.915451 (+564657us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:15.917531 (+ 2080us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":742984}
I20260504 13:56:16.242681 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3793fe8f1ec43ad809403c4593f5ef6" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:16.243279 9833 raft_consensus.cc:3060] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:16.251163 9931 ts_tablet_manager.cc:1434] T b37a738f2902402ca76e4edebcdba474 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.352s user 0.006s sys 0.000s
I20260504 13:56:16.253388 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_2d928cb3553f40bc8e825d0741ef3199"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:16.255764 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ed4f5ea230a413a8dcdbb661ce173b6" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:16.256300 9685 raft_consensus.cc:3060] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:16.257270 10213 raft_consensus.cc:493] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:16.257411 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 67) took 1516 ms. Trace:
I20260504 13:56:16.257565 9932 ts_tablet_manager.cc:1434] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.083s user 0.005s sys 0.000s
I20260504 13:56:16.257697 9944 rpcz_store.cc:276] 0504 13:56:14.740844 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:14.741442 (+ 598us) service_pool.cc:224] Handling call
0504 13:56:15.451824 (+710382us) ts_tablet_manager.cc:1151] Deleting tablet 0b7a71dbcc694e1a97019188cfb2c672
0504 13:56:15.451877 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:15.972651 (+520774us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:16.178823 (+206172us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:16.257355 (+ 78532us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43249,"tablet-delete.queue_time_us":709841}
I20260504 13:56:16.259367 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 0b7a71dbcc694e1a97019188cfb2c672 (table default.Table_6aecd26cb4c04baea5252f5dfbb75663 [id=0649fc4627314bca8b51d0e1f5022e6d]) successfully deleted
W20260504 13:56:16.255105 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d928cb3553f40bc8e825d0741ef3199 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:16.263509 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de639fda874248f0853e74ee60aaf758" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:16.257702 10213 raft_consensus.cc:515] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.265822 10213 leader_election.cc:290] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:16.267238 9663 tablet_service.cc:1558] Processing DeleteTablet for tablet 91371bf848f5465ea2db0c090583382d with delete_type TABLET_DATA_DELETED (replaced by tablet 60d3cc59ffe947fbb57b7d3301ccbac1) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:16.271598 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69ed77782cd34274932957106ce26766" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:16.271843 9944 tablet_replica.cc:333] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:16.272557 9944 raft_consensus.cc:2243] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Raft consensus shutting down.
W20260504 13:56:16.272625 9620 leader_election.cc:343] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:16.274569 9620 leader_election.cc:343] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:16.278581 9944 pending_rounds.cc:70] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6: Trying to abort 1 pending ops.
I20260504 13:56:16.278857 9944 pending_rounds.cc:77] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290580189220864 op_type: NO_OP noop_request { }
I20260504 13:56:16.280710 9944 raft_consensus.cc:2889] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:16.281080 9944 raft_consensus.cc:2272] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:16.282187 10213 consensus_queue.cc:1048] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:16.283073 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_bdb22fcd488647e0bfed4170cb70365d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:16.283813 9932 tablet_bootstrap.cc:492] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:16.284794 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdb22fcd488647e0bfed4170cb70365d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:16.285954 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 68) took 2349 ms. Trace:
I20260504 13:56:16.286548 9931 tablet_bootstrap.cc:492] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:16.286373 10002 rpcz_store.cc:276] 0504 13:56:13.936452 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:13.939154 (+ 2702us) service_pool.cc:224] Handling call
0504 13:56:16.285677 (+2346523us) ts_tablet_manager.cc:1151] Deleting tablet 52ff55bef5e54353ab9ca09130ab1e0e
0504 13:56:16.285720 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:16.285880 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2345892}
I20260504 13:56:16.289532 9944 ts_tablet_manager.cc:1916] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:16.290058 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 70) took 2282 ms. Trace:
W20260504 13:56:16.281525 9474 consensus_peers.cc:597] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:16.290669 10002 rpcz_store.cc:276] 0504 13:56:14.007374 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:14.007499 (+ 125us) service_pool.cc:224] Handling call
0504 13:56:16.289686 (+2282187us) ts_tablet_manager.cc:1151] Deleting tablet cdb86f6370bf4c5f9c488d85efb67ef7
0504 13:56:16.289725 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:16.290010 (+ 285us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2281569}
I20260504 13:56:16.291780 10002 tablet_replica.cc:333] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
W20260504 13:56:16.290572 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 259770us
I20260504 13:56:16.292505 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3793fe8f1ec43ad809403c4593f5ef6" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:16.293524 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de639fda874248f0853e74ee60aaf758" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:16.294016 9539 raft_consensus.cc:3060] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:16.297751 10002 raft_consensus.cc:2243] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:16.299539 10002 pending_rounds.cc:70] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00: Trying to abort 1 pending ops.
I20260504 13:56:16.299422 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ed4f5ea230a413a8dcdbb661ce173b6" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:16.313596 10002 pending_rounds.cc:77] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290585471139840 op_type: NO_OP noop_request { }
I20260504 13:56:16.319829 10002 raft_consensus.cc:2889] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:16.322614 9932 tablet_bootstrap.cc:654] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:16.322613 10003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 75) took 1583 ms. Trace:
I20260504 13:56:16.323480 10002 raft_consensus.cc:2272] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:16.323487 10003 rpcz_store.cc:276] 0504 13:56:14.739546 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:14.739701 (+ 155us) service_pool.cc:224] Handling call
0504 13:56:15.979496 (+1239795us) ts_tablet_manager.cc:1151] Deleting tablet 0b7a71dbcc694e1a97019188cfb2c672
0504 13:56:15.979536 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:16.071581 (+ 92045us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:16.223668 (+152087us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:16.322540 (+ 98872us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1239219}
I20260504 13:56:16.325999 10003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 80) took 1220 ms. Trace:
I20260504 13:56:16.327811 10003 rpcz_store.cc:276] 0504 13:56:15.105802 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:15.105923 (+ 121us) service_pool.cc:224] Handling call
0504 13:56:16.325579 (+1219656us) ts_tablet_manager.cc:1151] Deleting tablet 07a428049fdb4a68aae729400e2d9674
0504 13:56:16.325640 (+ 61us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:16.325937 (+ 297us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1219127}
I20260504 13:56:16.330416 10003 tablet_replica.cc:333] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:16.332849 10003 raft_consensus.cc:2243] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:16.333693 10003 raft_consensus.cc:2272] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:16.335925 10003 ts_tablet_manager.cc:1916] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:16.339857 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69ed77782cd34274932957106ce26766" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:16.282163 9536 raft_consensus.cc:2468] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
W20260504 13:56:16.347141 9618 leader_election.cc:343] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:16.347712 9618 leader_election.cc:304] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:16.348534 10095 raft_consensus.cc:2749] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:16.348949 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 46444.3us
W20260504 13:56:16.350163 9474 leader_election.cc:343] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:16.357404 9932 tablet_bootstrap.cc:492] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:16.357788 9932 ts_tablet_manager.cc:1403] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.074s user 0.009s sys 0.000s
I20260504 13:56:16.359570 9341 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_2aa57c2b774344218cd8fb99ed51879c" } new_table_name: "default.Table_64ff08822ec84a56b238186bea0eb82c" modify_external_catalogs: true
I20260504 13:56:16.339074 10002 ts_tablet_manager.cc:1916] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:16.370121 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 0b7a71dbcc694e1a97019188cfb2c672 (table default.Table_6aecd26cb4c04baea5252f5dfbb75663 [id=0649fc4627314bca8b51d0e1f5022e6d]) successfully deleted
W20260504 13:56:16.374058 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 07a428049fdb4a68aae729400e2d9674 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a428049fdb4a68aae729400e2d9674 already in progress: creating tablet
I20260504 13:56:16.394425 9344 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 972bff0372e9472082d3f5ed74a29057
I20260504 13:56:16.403963 9344 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_363d6634e42e4dd6a5686c1f0e401ef2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:16.405946 9344 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_363d6634e42e4dd6a5686c1f0e401ef2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:16.437042 10013 tablet_bootstrap.cc:654] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:16.444528 10194 consensus_meta.cc:287] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent flushing consensus metadata: real 0.593s user 0.006s sys 0.004s
I20260504 13:56:16.439801 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 03818e34ac0545be880c873cf82cd4ea (table default.Table_078d1859743841e985b4e0e1c4f61030 [id=89a6d18328c04d90828f3e51696592fa]) successfully deleted
W20260504 13:56:16.450444 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 52ff55bef5e54353ab9ca09130ab1e0e because the tablet was not found. No further retry: Not found: Tablet not found: 52ff55bef5e54353ab9ca09130ab1e0e
W20260504 13:56:16.452211 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet cdb86f6370bf4c5f9c488d85efb67ef7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cdb86f6370bf4c5f9c488d85efb67ef7 already in progress: creating tablet
W20260504 13:56:16.447291 9473 consensus_peers.cc:597] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b7a71dbcc694e1a97019188cfb2c672. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:16.458123 9517 tablet_service.cc:1558] Processing DeleteTablet for tablet 91371bf848f5465ea2db0c090583382d with delete_type TABLET_DATA_DELETED (replaced by tablet 60d3cc59ffe947fbb57b7d3301ccbac1) from {username='slave'} at 127.0.0.1:52642
W20260504 13:56:16.449360 9474 leader_election.cc:343] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:16.467594 9474 leader_election.cc:343] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:16.453763 10194 raft_consensus.cc:515] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.338011 9833 raft_consensus.cc:2468] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:16.484557 10194 leader_election.cc:290] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:16.494982 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fdc09a498be470ab04563c62caf7755" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:16.495558 9834 raft_consensus.cc:3060] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:16.498687 9931 tablet_bootstrap.cc:654] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:16.499827 10194 consensus_queue.cc:1048] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260504 13:56:16.504878 10200 consensus_meta.cc:287] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent flushing consensus metadata: real 0.580s user 0.005s sys 0.003s
I20260504 13:56:16.504742 9539 raft_consensus.cc:2468] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:16.505461 10200 raft_consensus.cc:515] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.507269 10200 leader_election.cc:290] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:16.508126 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b37a738f2902402ca76e4edebcdba474" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:16.509023 9831 raft_consensus.cc:3060] T b37a738f2902402ca76e4edebcdba474 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:16.510851 9834 raft_consensus.cc:2468] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:16.516955 9474 leader_election.cc:304] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:16.517884 10200 raft_consensus.cc:2804] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:16.518343 10200 raft_consensus.cc:697] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:16.519155 10200 consensus_queue.cc:237] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.546411 9931 tablet_bootstrap.cc:492] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:16.547056 9931 ts_tablet_manager.cc:1403] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.286s user 0.009s sys 0.003s
I20260504 13:56:16.549945 9931 raft_consensus.cc:359] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.553355 9931 raft_consensus.cc:385] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:16.554270 10228 tablet.cc:1724] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260504 13:56:16.563580 9685 raft_consensus.cc:2468] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:16.568387 10013 tablet_bootstrap.cc:492] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:16.568835 10013 ts_tablet_manager.cc:1403] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.370s user 0.018s sys 0.017s
W20260504 13:56:16.570283 9341 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0be795f04dd54cf19869c937b91d0ddf has no consensus state
I20260504 13:56:16.575160 9344 master_service.cc:954] ReplaceTablet: received request to replace tablet c34877a81d504f46a2f680c5d5ddb4f4 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:16.576050 9932 raft_consensus.cc:359] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.576668 9932 raft_consensus.cc:385] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:16.576912 9932 raft_consensus.cc:740] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:16.577414 9932 consensus_queue.cc:260] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.553679 9931 raft_consensus.cc:740] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:16.579897 9931 consensus_queue.cc:260] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.579479 10095 consensus_queue.cc:1048] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
W20260504 13:56:16.579612 9618 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 127533us
W20260504 13:56:16.581115 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 183493us
I20260504 13:56:16.596402 9473 leader_election.cc:304] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:16.597407 10200 raft_consensus.cc:2804] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:16.598093 10200 raft_consensus.cc:697] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:16.598186 10013 raft_consensus.cc:359] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.598759 10013 raft_consensus.cc:385] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:16.598941 10200 consensus_queue.cc:237] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.598989 10013 raft_consensus.cc:740] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:16.603583 10013 consensus_queue.cc:260] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.604139 9618 leader_election.cc:304] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:16.579135 9932 ts_tablet_manager.cc:1434] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.221s user 0.017s sys 0.009s
I20260504 13:56:16.611003 10013 ts_tablet_manager.cc:1434] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.042s user 0.001s sys 0.003s
I20260504 13:56:16.611788 10002 ts_tablet_manager.cc:1929] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:16.612155 10002 log.cc:1199] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/14b7f1e834d0439c89abb180bd13853d
I20260504 13:56:16.612819 10002 ts_tablet_manager.cc:1950] T 14b7f1e834d0439c89abb180bd13853d P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:16.612823 9341 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_47c9ed0b601146b687a576ec89c8b626" } modify_external_catalogs: true
I20260504 13:56:16.613346 9341 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_47c9ed0b601146b687a576ec89c8b626" } modify_external_catalogs: true
I20260504 13:56:16.611057 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_597b4ca3e37849d0aa9c86eed64d0614"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:16.609098 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fdc09a498be470ab04563c62caf7755" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:16.614598 9944 ts_tablet_manager.cc:1929] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:16.611794 10013 tablet_bootstrap.cc:492] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:16.615674 9684 raft_consensus.cc:3060] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:16.615828 10095 raft_consensus.cc:2804] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:16.616436 10095 raft_consensus.cc:697] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
W20260504 13:56:16.616494 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_597b4ca3e37849d0aa9c86eed64d0614 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:16.617168 10095 consensus_queue.cc:237] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.621758 9932 tablet_bootstrap.cc:492] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:16.625876 9944 log.cc:1199] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/03818e34ac0545be880c873cf82cd4ea
I20260504 13:56:16.626627 9944 ts_tablet_manager.cc:1950] T 03818e34ac0545be880c873cf82cd4ea P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:16.629166 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b37a738f2902402ca76e4edebcdba474" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:16.629671 9685 raft_consensus.cc:3060] T b37a738f2902402ca76e4edebcdba474 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:16.630223 10194 consensus_queue.cc:1048] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:16.634284 10013 tablet_bootstrap.cc:654] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:16.635035 9931 ts_tablet_manager.cc:1434] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.088s user 0.005s sys 0.000s
I20260504 13:56:16.651587 10013 tablet_bootstrap.cc:492] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:16.651971 10013 ts_tablet_manager.cc:1403] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.000s
I20260504 13:56:16.664856 9338 master_service.cc:954] ReplaceTablet: received request to replace tablet 39f080a0296b44988e8e51e68453a551 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:16.668152 9932 tablet_bootstrap.cc:654] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:16.669580 10013 raft_consensus.cc:359] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.670172 10013 raft_consensus.cc:385] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:16.670426 10013 raft_consensus.cc:740] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:16.670986 10013 consensus_queue.cc:260] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.676497 10013 ts_tablet_manager.cc:1434] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20260504 13:56:16.677313 10013 tablet_bootstrap.cc:492] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:16.689097 10013 tablet_bootstrap.cc:654] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:16.715389 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 69) took 2711 ms. Trace:
I20260504 13:56:16.715718 10002 rpcz_store.cc:276] 0504 13:56:14.003926 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:14.035264 (+ 31338us) service_pool.cc:224] Handling call
0504 13:56:16.291609 (+2256345us) ts_tablet_manager.cc:1151] Deleting tablet 14b7f1e834d0439c89abb180bd13853d
0504 13:56:16.291657 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:16.516323 (+224666us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:16.611667 (+ 95344us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:16.715319 (+103652us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2255465}
I20260504 13:56:16.716734 10002 tablet_replica.cc:333] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:16.717587 10002 raft_consensus.cc:2243] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:16.718122 10002 raft_consensus.cc:2272] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:16.721844 10002 ts_tablet_manager.cc:1916] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:16.726882 9684 raft_consensus.cc:2468] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:16.731962 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 68) took 1990 ms. Trace:
I20260504 13:56:16.732311 9944 rpcz_store.cc:276] 0504 13:56:14.741168 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:14.742077 (+ 909us) service_pool.cc:224] Handling call
0504 13:56:16.258720 (+1516643us) ts_tablet_manager.cc:1151] Deleting tablet 03818e34ac0545be880c873cf82cd4ea
0504 13:56:16.271727 (+ 13007us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:16.560241 (+288514us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:16.614487 (+ 54246us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:16.731906 (+117419us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4947,"tablet-delete.queue_time_us":1516147}
I20260504 13:56:16.740187 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 14b7f1e834d0439c89abb180bd13853d (table default.Table_a0a1002687744b879e47ae4fd701d903 [id=2d45d703594b416893908650027e94b3]) successfully deleted
I20260504 13:56:16.735194 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 03818e34ac0545be880c873cf82cd4ea (table default.Table_078d1859743841e985b4e0e1c4f61030 [id=89a6d18328c04d90828f3e51696592fa]) successfully deleted
I20260504 13:56:16.736305 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 73) took 1639 ms. Trace:
I20260504 13:56:16.748157 9944 rpcz_store.cc:276] 0504 13:56:15.097565 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:15.097724 (+ 159us) service_pool.cc:224] Handling call
0504 13:56:16.733353 (+1635629us) ts_tablet_manager.cc:1151] Deleting tablet 07a428049fdb4a68aae729400e2d9674
0504 13:56:16.733405 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:16.736257 (+ 2852us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1635078}
W20260504 13:56:16.750968 9329 catalog_manager.cc:4982] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 07a428049fdb4a68aae729400e2d9674 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a428049fdb4a68aae729400e2d9674 already in progress: creating tablet
I20260504 13:56:16.751220 9944 tablet_replica.cc:333] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:16.751868 9944 raft_consensus.cc:2243] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:16.753847 10013 tablet_bootstrap.cc:492] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:16.754242 10013 ts_tablet_manager.cc:1403] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.077s user 0.014s sys 0.007s
I20260504 13:56:16.756567 10013 raft_consensus.cc:359] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.764600 9944 raft_consensus.cc:2272] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:16.769419 9944 ts_tablet_manager.cc:1916] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:16.775075 9620 leader_election.cc:304] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:16.776368 10213 raft_consensus.cc:2804] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:16.776806 10213 raft_consensus.cc:697] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:16.777582 10213 consensus_queue.cc:237] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:16.780954 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 389268us
I20260504 13:56:16.782963 9809 tablet_service.cc:1558] Processing DeleteTablet for tablet 91371bf848f5465ea2db0c090583382d with delete_type TABLET_DATA_DELETED (replaced by tablet 60d3cc59ffe947fbb57b7d3301ccbac1) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:16.791056 9831 raft_consensus.cc:2468] T b37a738f2902402ca76e4edebcdba474 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
W20260504 13:56:16.792095 9474 consensus_peers.cc:597] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:16.799290 9474 leader_election.cc:304] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:16.800014 10200 raft_consensus.cc:2804] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:16.800472 10200 raft_consensus.cc:697] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:16.801297 10200 consensus_queue.cc:237] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.803289 9685 raft_consensus.cc:2468] T b37a738f2902402ca76e4edebcdba474 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:16.636119 9931 tablet_bootstrap.cc:492] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:16.832254 10003 ts_tablet_manager.cc:1929] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:16.832764 10003 log.cc:1199] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/5625163f6add4a20b1828905d90b5486
I20260504 13:56:16.833642 10003 ts_tablet_manager.cc:1950] T 5625163f6add4a20b1828905d90b5486 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:16.836771 10003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 82) took 1476 ms. Trace:
I20260504 13:56:16.838987 10003 rpcz_store.cc:276] 0504 13:56:15.360194 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:15.360316 (+ 122us) service_pool.cc:224] Handling call
0504 13:56:16.330285 (+969969us) ts_tablet_manager.cc:1151] Deleting tablet 5625163f6add4a20b1828905d90b5486
0504 13:56:16.330321 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:16.639542 (+309221us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:16.831843 (+192301us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:16.836682 (+ 4839us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":969404}
I20260504 13:56:16.840238 10003 tablet_replica.cc:333] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:16.841102 10003 raft_consensus.cc:2243] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:16.841691 10003 raft_consensus.cc:2272] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:16.843942 10003 ts_tablet_manager.cc:1916] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:16.844133 9932 tablet_bootstrap.cc:492] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:16.847501 9932 ts_tablet_manager.cc:1403] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.226s user 0.013s sys 0.012s
I20260504 13:56:16.849850 9932 raft_consensus.cc:359] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.850433 9932 raft_consensus.cc:385] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:16.850664 9932 raft_consensus.cc:740] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
W20260504 13:56:16.854714 9328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 111617us
W20260504 13:56:16.857784 9343 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:55533 (request call id 29) took 1424 ms (client timeout 1000 ms). Trace:
W20260504 13:56:16.859311 9343 rpcz_store.cc:269] 0504 13:56:15.433665 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:15.433786 (+ 121us) service_pool.cc:224] Handling call
0504 13:56:15.446332 (+ 12546us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:16.857719 (+1411387us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":2,"spinlock_wait_cycles":25113344}
I20260504 13:56:16.865231 10228 tablet.cc:1724] T e0e3b36e1f654a3a8df51480801dabd7 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:16.865105 9932 consensus_queue.cc:260] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.868556 9932 ts_tablet_manager.cc:1434] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.021s user 0.004s sys 0.001s
I20260504 13:56:16.876389 9932 tablet_bootstrap.cc:492] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:16.879977 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 90030.9us
I20260504 13:56:16.881634 9341 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet bd8cd1caaeb249e3bafee7f1f0d4df2f
I20260504 13:56:16.884572 9663 tablet_service.cc:1558] Processing DeleteTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:16 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:16.885741 9514 tablet_service.cc:1558] Processing DeleteTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:16 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:16.892648 9932 tablet_bootstrap.cc:654] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:16.893088 10213 raft_consensus.cc:493] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:16.893674 10213 raft_consensus.cc:515] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.896674 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "baf07a970a7c4a35b3c849b06d6c88d0" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:16.898447 9618 leader_election.cc:343] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:16.899966 10213 leader_election.cc:290] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:16.907524 9944 ts_tablet_manager.cc:1929] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:16.757220 10013 raft_consensus.cc:385] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:16.907866 9944 log.cc:1199] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/5625163f6add4a20b1828905d90b5486
I20260504 13:56:16.907848 10013 raft_consensus.cc:740] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:16.908564 9944 ts_tablet_manager.cc:1950] T 5625163f6add4a20b1828905d90b5486 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:16.908455 10013 consensus_queue.cc:260] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.920806 10095 raft_consensus.cc:493] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:16.921461 10095 raft_consensus.cc:515] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.927534 9344 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet c34877a81d504f46a2f680c5d5ddb4f4
I20260504 13:56:16.928229 10095 leader_election.cc:290] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:16.928673 10095 raft_consensus.cc:493] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:16.929241 9514 tablet_service.cc:1558] Processing DeleteTablet for tablet c34877a81d504f46a2f680c5d5ddb4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 359b0fa01a1648a7bef6f97960a7b9a8) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:16.929543 9663 tablet_service.cc:1558] Processing DeleteTablet for tablet c34877a81d504f46a2f680c5d5ddb4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 359b0fa01a1648a7bef6f97960a7b9a8) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:16.929718 9344 catalog_manager.cc:6741] ReplaceTablet: tablet c34877a81d504f46a2f680c5d5ddb4f4 deleted and replaced by tablet 359b0fa01a1648a7bef6f97960a7b9a8
I20260504 13:56:16.930204 10095 raft_consensus.cc:515] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:16.930625 9344 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_24dab2dd1f45402786d80cba80bbad82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:16.932228 9344 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24dab2dd1f45402786d80cba80bbad82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:16.932489 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b193e958b4414aad29dd2bcee1a61b" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:16.933182 9539 raft_consensus.cc:2468] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:16.936415 9618 leader_election.cc:304] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:16.937179 10215 raft_consensus.cc:2804] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:16.937578 10215 raft_consensus.cc:493] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:16.937909 10215 raft_consensus.cc:3060] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:16.944327 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7ab3b577b74a9c9a40cb15c94f8aa1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:16.945742 9618 leader_election.cc:343] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:16.946355 10095 leader_election.cc:290] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
W20260504 13:56:16.957549 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 173492us
I20260504 13:56:16.962738 9344 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_a8d8f096360b4b5281f5124fe966098a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:16.966877 10254 tablet.cc:1724] T e0e3b36e1f654a3a8df51480801dabd7 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:16.967947 9336 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:16.968912 9336 catalog_manager.cc:5671] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:16.971112 9336 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet e0e3b36e1f654a3a8df51480801dabd7 (table default.Table_9e9ef3353dcf4b1fadaa2f51bc8f7041 [id=8dcabd7d67334598926f927a62e630d6]). Expected version 2 got 0
I20260504 13:56:16.977182 9401 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet bd8cd1caaeb249e3bafee7f1f0d4df2f
I20260504 13:56:16.989451 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 76) took 1636 ms. Trace:
I20260504 13:56:16.989893 9944 rpcz_store.cc:276] 0504 13:56:15.352762 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:15.357115 (+ 4353us) service_pool.cc:224] Handling call
0504 13:56:16.751056 (+1393941us) ts_tablet_manager.cc:1151] Deleting tablet 5625163f6add4a20b1828905d90b5486
0504 13:56:16.751103 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:16.805158 (+ 54055us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:16.907404 (+102246us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:16.989340 (+ 81936us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2770,"tablet-delete.queue_time_us":1393416}
I20260504 13:56:16.992404 9944 tablet_replica.cc:333] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:16.993335 9944 raft_consensus.cc:2243] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:16.994383 9944 raft_consensus.cc:2272] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:16.996894 9944 ts_tablet_manager.cc:1916] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:16.991529 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 5625163f6add4a20b1828905d90b5486 (table default.Table_56891f8543a84e859c39b381625a9daa [id=7b3ef250728944d9998ef2df032f8819]) successfully deleted
I20260504 13:56:17.007865 9514 tablet_service.cc:1558] Processing DeleteTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:16 UTC) from {username='slave'} at 127.0.0.1:52642
W20260504 13:56:16.998896 9344 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8d8f096360b4b5281f5124fe966098a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:17.012578 10013 ts_tablet_manager.cc:1434] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.258s user 0.006s sys 0.001s
I20260504 13:56:17.013263 9663 tablet_service.cc:1558] Processing DeleteTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:16 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:17.015569 10215 raft_consensus.cc:515] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.018853 9932 tablet_bootstrap.cc:492] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:17.019243 9932 ts_tablet_manager.cc:1403] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.143s user 0.019s sys 0.026s
I20260504 13:56:17.021486 9932 raft_consensus.cc:359] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.022058 9932 raft_consensus.cc:385] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:17.022284 9932 raft_consensus.cc:740] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:17.022596 10215 leader_election.cc:290] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:17.021037 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b193e958b4414aad29dd2bcee1a61b" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:17.023425 9539 raft_consensus.cc:3060] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:17.023808 10013 tablet_bootstrap.cc:492] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:17.039191 10013 tablet_bootstrap.cc:654] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:17.043561 9338 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 39f080a0296b44988e8e51e68453a551
I20260504 13:56:17.044785 9338 catalog_manager.cc:6741] ReplaceTablet: tablet 39f080a0296b44988e8e51e68453a551 deleted and replaced by tablet 274bb204041c4e1085395c7bd938aaf1
I20260504 13:56:17.036635 9932 consensus_queue.cc:260] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.046329 9514 tablet_service.cc:1558] Processing DeleteTablet for tablet 39f080a0296b44988e8e51e68453a551 with delete_type TABLET_DATA_DELETED (replaced by tablet 274bb204041c4e1085395c7bd938aaf1) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:17.047623 9663 tablet_service.cc:1558] Processing DeleteTablet for tablet 39f080a0296b44988e8e51e68453a551 with delete_type TABLET_DATA_DELETED (replaced by tablet 274bb204041c4e1085395c7bd938aaf1) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:17.048115 9932 ts_tablet_manager.cc:1434] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.029s user 0.005s sys 0.001s
I20260504 13:56:17.049892 9342 catalog_manager.cc:5671] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:17.051699 9342 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:17.052189 9932 tablet_bootstrap.cc:492] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:17.055801 9342 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet e0e3b36e1f654a3a8df51480801dabd7 (table default.Table_9e9ef3353dcf4b1fadaa2f51bc8f7041 [id=8dcabd7d67334598926f927a62e630d6]). Expected version 2 got 0
I20260504 13:56:17.082574 9932 tablet_bootstrap.cc:654] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:17.083387 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "baf07a970a7c4a35b3c849b06d6c88d0" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:17.084403 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b193e958b4414aad29dd2bcee1a61b" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:17.084926 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b193e958b4414aad29dd2bcee1a61b" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:17.085703 9620 leader_election.cc:343] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:17.086215 9620 leader_election.cc:304] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:17.086421 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7ab3b577b74a9c9a40cb15c94f8aa1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:17.087984 10213 raft_consensus.cc:2749] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:17.088991 10246 raft_consensus.cc:493] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:17.089517 10246 raft_consensus.cc:515] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.094269 9932 tablet_bootstrap.cc:492] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:17.094661 9932 ts_tablet_manager.cc:1403] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.003s
W20260504 13:56:17.098732 9330 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet afee197774db411d97468913a207703a (table default.Table_91c2d3ed5c5c42758c87f1f726af88c0 [id=a2697dba5f38435c81dc52316712917d], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet afee197774db411d97468913a207703a consensus state has no leader
W20260504 13:56:17.103332 9620 leader_election.cc:343] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:17.104046 9620 leader_election.cc:343] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:17.104857 9620 leader_election.cc:343] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:17.105190 9620 leader_election.cc:304] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:17.105973 10213 raft_consensus.cc:2749] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:17.108270 9932 raft_consensus.cc:359] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.108873 9932 raft_consensus.cc:385] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:17.109205 9932 raft_consensus.cc:740] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:17.109838 9932 consensus_queue.cc:260] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.119570 9539 raft_consensus.cc:2468] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:17.121130 10002 ts_tablet_manager.cc:1929] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:17.121408 10246 leader_election.cc:290] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:17.121657 10002 log.cc:1199] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/b4dca0cadd5e400eab1365c03a8de4f4
I20260504 13:56:17.122654 10002 ts_tablet_manager.cc:1950] T b4dca0cadd5e400eab1365c03a8de4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:17.122943 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ed4f5ea230a413a8dcdbb661ce173b6" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:17.123644 9685 raft_consensus.cc:2393] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:17.126359 9932 ts_tablet_manager.cc:1434] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.031s user 0.001s sys 0.005s
I20260504 13:56:17.127187 9932 tablet_bootstrap.cc:492] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:17.129012 9618 leader_election.cc:304] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:17.130370 10213 raft_consensus.cc:2804] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:17.131000 10213 raft_consensus.cc:697] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:17.132082 10213 consensus_queue.cc:237] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.149760 9932 tablet_bootstrap.cc:654] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:17.155134 9931 tablet_bootstrap.cc:654] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:17.164767 9932 tablet_bootstrap.cc:492] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:17.165184 9932 ts_tablet_manager.cc:1403] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.003s
I20260504 13:56:17.167614 9932 raft_consensus.cc:359] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.168196 9932 raft_consensus.cc:385] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:17.168419 9932 raft_consensus.cc:740] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:17.178485 9932 consensus_queue.cc:260] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.184371 10215 raft_consensus.cc:493] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:17.184840 10215 raft_consensus.cc:515] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.186623 10215 leader_election.cc:290] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:17.189234 9932 ts_tablet_manager.cc:1434] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260504 13:56:17.190125 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afee197774db411d97468913a207703a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:17.191568 9618 leader_election.cc:343] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:17.193290 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afee197774db411d97468913a207703a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:17.194664 9620 leader_election.cc:343] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:17.194969 9932 tablet_bootstrap.cc:492] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:17.195122 9620 leader_election.cc:304] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:17.196090 10215 raft_consensus.cc:2749] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:17.199587 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 5625163f6add4a20b1828905d90b5486 (table default.Table_56891f8543a84e859c39b381625a9daa [id=7b3ef250728944d9998ef2df032f8819]) successfully deleted
I20260504 13:56:17.202085 9338 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_895e961d6c1a45e688ad4e0ca752bae9" } new_table_name: "default.Table_11d8ab61a6044154bb599875a0bc9004" modify_external_catalogs: true
I20260504 13:56:17.210253 9932 tablet_bootstrap.cc:654] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:17.217715 9514 tablet_service.cc:1511] Processing CreateTablet for tablet acd5f7d3f7c545d18fe19b64e4bc1de6 (DEFAULT_TABLE table=default.Table_f7b06aadc0024855b08b56cad8c7b6a3 [id=4efd9dd69847446d9f05700d3bd3f914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.218957 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acd5f7d3f7c545d18fe19b64e4bc1de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.221985 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 71) took 3108 ms. Trace:
I20260504 13:56:17.222332 10002 rpcz_store.cc:276] 0504 13:56:14.113361 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:14.114211 (+ 850us) service_pool.cc:224] Handling call
0504 13:56:16.716559 (+2602348us) ts_tablet_manager.cc:1151] Deleting tablet b4dca0cadd5e400eab1365c03a8de4f4
0504 13:56:16.716595 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:16.950118 (+233523us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:17.120965 (+170847us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:17.221935 (+100970us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4864,"tablet-delete.queue_time_us":2601700}
I20260504 13:56:17.225776 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet b4dca0cadd5e400eab1365c03a8de4f4 (table default.Table_15bf9089e68d4882838fe4f1c65156ef [id=3814e135130a4213a46df3cd93eca49e]) successfully deleted
I20260504 13:56:17.226797 10002 tablet_replica.cc:333] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:17.227600 10002 raft_consensus.cc:2243] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:17.227836 9932 tablet_bootstrap.cc:492] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:17.229557 9663 tablet_service.cc:1511] Processing CreateTablet for tablet acd5f7d3f7c545d18fe19b64e4bc1de6 (DEFAULT_TABLE table=default.Table_f7b06aadc0024855b08b56cad8c7b6a3 [id=4efd9dd69847446d9f05700d3bd3f914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.230406 9932 ts_tablet_manager.cc:1403] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.003s
I20260504 13:56:17.232617 9932 raft_consensus.cc:359] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.233193 9932 raft_consensus.cc:385] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:17.233415 9932 raft_consensus.cc:740] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:17.236279 9515 tablet_service.cc:1511] Processing CreateTablet for tablet 0be795f04dd54cf19869c937b91d0ddf (DEFAULT_TABLE table=default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.237854 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0be795f04dd54cf19869c937b91d0ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.239732 10200 raft_consensus.cc:493] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:17.240214 10200 raft_consensus.cc:515] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.242039 10200 leader_election.cc:290] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:17.243569 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69ed77782cd34274932957106ce26766" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:17.244127 9685 raft_consensus.cc:2468] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:17.245648 9932 consensus_queue.cc:260] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.246763 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69ed77782cd34274932957106ce26766" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:17.247784 9473 leader_election.cc:304] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
W20260504 13:56:17.248585 9474 leader_election.cc:343] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:17.249607 10259 tablet.cc:1724] T e0e3b36e1f654a3a8df51480801dabd7 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:17.251415 10200 raft_consensus.cc:2804] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:17.252451 9932 ts_tablet_manager.cc:1434] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20260504 13:56:17.253214 9932 tablet_bootstrap.cc:492] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:17.251909 10200 raft_consensus.cc:493] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:17.254179 10200 raft_consensus.cc:3060] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:17.228607 10002 raft_consensus.cc:2272] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:17.260026 9809 tablet_service.cc:1558] Processing DeleteTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:16 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:17.261098 9809 tablet_service.cc:1558] Processing DeleteTablet for tablet c34877a81d504f46a2f680c5d5ddb4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 359b0fa01a1648a7bef6f97960a7b9a8) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:17.261831 10002 ts_tablet_manager.cc:1916] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:17.262027 9809 tablet_service.cc:1558] Processing DeleteTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:16 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:17.262385 9932 tablet_bootstrap.cc:654] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:17.265920 10013 tablet_bootstrap.cc:492] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:17.266294 10013 ts_tablet_manager.cc:1403] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.243s user 0.020s sys 0.037s
I20260504 13:56:17.268471 10013 raft_consensus.cc:359] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.269028 10013 raft_consensus.cc:385] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:17.269304 10013 raft_consensus.cc:740] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:17.273931 10013 consensus_queue.cc:260] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.275235 9809 tablet_service.cc:1511] Processing CreateTablet for tablet acd5f7d3f7c545d18fe19b64e4bc1de6 (DEFAULT_TABLE table=default.Table_f7b06aadc0024855b08b56cad8c7b6a3 [id=4efd9dd69847446d9f05700d3bd3f914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.276618 9809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acd5f7d3f7c545d18fe19b64e4bc1de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.282266 10013 ts_tablet_manager.cc:1434] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260504 13:56:17.286744 10013 tablet_bootstrap.cc:492] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:17.287387 9931 tablet_bootstrap.cc:492] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:17.287823 9931 ts_tablet_manager.cc:1403] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.652s user 0.017s sys 0.000s
I20260504 13:56:17.290356 9931 raft_consensus.cc:359] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.301205 10013 tablet_bootstrap.cc:654] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:17.308881 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ed4f5ea230a413a8dcdbb661ce173b6" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:17.326943 9944 ts_tablet_manager.cc:1929] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:17.327541 9944 log.cc:1199] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/91371bf848f5465ea2db0c090583382d
I20260504 13:56:17.328533 9944 ts_tablet_manager.cc:1950] T 91371bf848f5465ea2db0c090583382d P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:17.329520 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acd5f7d3f7c545d18fe19b64e4bc1de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.349512 10215 raft_consensus.cc:493] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:17.349928 10215 raft_consensus.cc:515] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.351640 10215 leader_election.cc:290] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:17.353806 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b50573310a04f3694eeab2fef0fb540" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:17.354839 9618 leader_election.cc:343] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:17.359872 9665 tablet_service.cc:1511] Processing CreateTablet for tablet 0be795f04dd54cf19869c937b91d0ddf (DEFAULT_TABLE table=default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.301246 9813 tablet_service.cc:1558] Processing DeleteTablet for tablet 39f080a0296b44988e8e51e68453a551 with delete_type TABLET_DATA_DELETED (replaced by tablet 274bb204041c4e1085395c7bd938aaf1) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:17.302867 9812 tablet_service.cc:1511] Processing CreateTablet for tablet 0be795f04dd54cf19869c937b91d0ddf (DEFAULT_TABLE table=default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.355731 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b50573310a04f3694eeab2fef0fb540" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:17.376741 9620 leader_election.cc:343] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:17.377184 9620 leader_election.cc:304] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:17.377948 10215 raft_consensus.cc:2749] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:17.382846 9931 raft_consensus.cc:385] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:17.383744 9931 raft_consensus.cc:740] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:17.387351 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0be795f04dd54cf19869c937b91d0ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.420135 9931 consensus_queue.cc:260] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.373876 9767 leader_election.cc:304] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:17.426288 10246 raft_consensus.cc:3060] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:17.426932 10003 ts_tablet_manager.cc:1929] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:17.427477 10003 log.cc:1199] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/91371bf848f5465ea2db0c090583382d
I20260504 13:56:17.428467 10003 ts_tablet_manager.cc:1950] T 91371bf848f5465ea2db0c090583382d P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:17.429486 10200 raft_consensus.cc:515] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.433858 10195 consensus_queue.cc:1048] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:17.434832 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 79) took 1178 ms. Trace:
I20260504 13:56:17.435513 10195 raft_consensus.cc:493] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:17.435218 9944 rpcz_store.cc:276] 0504 13:56:16.256234 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:16.267067 (+ 10833us) service_pool.cc:224] Handling call
0504 13:56:16.992196 (+725129us) ts_tablet_manager.cc:1151] Deleting tablet 91371bf848f5465ea2db0c090583382d
0504 13:56:16.992253 (+ 57us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:17.116499 (+124246us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:17.326753 (+210254us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:17.434770 (+108017us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":724508}
I20260504 13:56:17.436015 10195 raft_consensus.cc:515] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.438977 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "baf07a970a7c4a35b3c849b06d6c88d0" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:17.438593 10195 leader_election.cc:290] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
W20260504 13:56:17.440809 9474 leader_election.cc:343] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:17.452315 10200 leader_election.cc:290] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:17.455145 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69ed77782cd34274932957106ce26766" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:17.455982 9829 raft_consensus.cc:3060] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:17.437503 9944 tablet_replica.cc:333] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
W20260504 13:56:17.463544 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 89752.2us
I20260504 13:56:17.459120 9944 raft_consensus.cc:2243] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:17.465202 9944 raft_consensus.cc:2272] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:17.468001 9944 ts_tablet_manager.cc:1916] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:17.469808 10200 raft_consensus.cc:493] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:17.470427 10200 raft_consensus.cc:515] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:17.471267 9338 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8e4239942f34fb081a026a35740e465 consensus state has no leader
I20260504 13:56:17.475795 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a43cdb767fd14ea09b4db4d2881b69aa" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:17.477721 9474 leader_election.cc:343] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:17.484736 10013 tablet_bootstrap.cc:492] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:17.485153 10013 ts_tablet_manager.cc:1403] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.202s user 0.011s sys 0.037s
I20260504 13:56:17.487280 9812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0be795f04dd54cf19869c937b91d0ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.491430 10200 leader_election.cc:290] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:17.492126 9344 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_7db59fa280b6442b99cf24677b57df83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:17.493144 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_44cfda2157f6473cb8b53b4940f279cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:17.494105 9344 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7db59fa280b6442b99cf24677b57df83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:17.493392 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 91371bf848f5465ea2db0c090583382d (table default.Table_28052c777ee944b79b802d28b415c24e [id=994b65b95f66440283f852cd9dd9ae0c]) successfully deleted
W20260504 13:56:17.495280 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f because the tablet was not found. No further retry: Not found: Tablet not found: bd8cd1caaeb249e3bafee7f1f0d4df2f
W20260504 13:56:17.495911 9329 catalog_manager.cc:4982] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet c34877a81d504f46a2f680c5d5ddb4f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c34877a81d504f46a2f680c5d5ddb4f4 already in progress: creating tablet
W20260504 13:56:17.496515 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f because the tablet was not found. No further retry: Not found: Tablet not found: bd8cd1caaeb249e3bafee7f1f0d4df2f
W20260504 13:56:17.494509 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44cfda2157f6473cb8b53b4940f279cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:17.482829 9932 tablet_bootstrap.cc:492] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:17.499660 9932 ts_tablet_manager.cc:1403] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.247s user 0.014s sys 0.044s
I20260504 13:56:17.505296 9932 raft_consensus.cc:359] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.505914 9932 raft_consensus.cc:385] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:17.506162 9932 raft_consensus.cc:740] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:17.503791 10215 raft_consensus.cc:493] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:17.506628 10215 raft_consensus.cc:515] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.506736 9932 consensus_queue.cc:260] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.509025 9343 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_1f5a66e30f35482dab7c556daebc87c9" } modify_external_catalogs: true
I20260504 13:56:17.509466 10215 leader_election.cc:290] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:17.509666 9343 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_1f5a66e30f35482dab7c556daebc87c9" } modify_external_catalogs: true
I20260504 13:56:17.510354 9932 ts_tablet_manager.cc:1434] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260504 13:56:17.512054 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c43ff1d759a40d68344f25f066416ac" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:17.514045 9618 leader_election.cc:343] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:17.514464 9932 tablet_bootstrap.cc:492] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:17.516853 9341 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:17.521349 9931 ts_tablet_manager.cc:1434] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.233s user 0.006s sys 0.000s
I20260504 13:56:17.522543 9931 tablet_bootstrap.cc:492] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:17.523276 10246 raft_consensus.cc:2749] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:17.523427 9932 tablet_bootstrap.cc:654] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:17.528913 9341 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet e0e3b36e1f654a3a8df51480801dabd7 (table default.Table_9e9ef3353dcf4b1fadaa2f51bc8f7041 [id=8dcabd7d67334598926f927a62e630d6]). Expected version 2 got 0
I20260504 13:56:17.530648 10003 tablet_replica.cc:333] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:17.546331 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "baf07a970a7c4a35b3c849b06d6c88d0" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:17.547147 9685 raft_consensus.cc:2468] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:17.548552 9338 master_service.cc:954] ReplaceTablet: received request to replace tablet a43cdb767fd14ea09b4db4d2881b69aa from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:17.549126 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69ed77782cd34274932957106ce26766" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:17.549510 9344 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_b59e5e285d50401e94d51afeb4cbc500"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:17.549830 9685 raft_consensus.cc:3060] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:17.551074 9344 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b59e5e285d50401e94d51afeb4cbc500 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:17.558243 9517 tablet_service.cc:1511] Processing CreateTablet for tablet ab4c185332a24d96bc3d0fddc0d20693 (DEFAULT_TABLE table=default.Table_af058f50341e4f478c970f8cb52bf0d7 [id=57ccd7218d454c8c908fe06b798bd83b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.558883 9515 tablet_service.cc:1511] Processing CreateTablet for tablet b13303d2a14c4187b82da9f25ce9753e (DEFAULT_TABLE table=default.Table_b32f665dddb043769e07076397f84a73 [id=f227576f105c4b1fbd459b3638ce684b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.565084 9813 tablet_service.cc:1511] Processing CreateTablet for tablet b13303d2a14c4187b82da9f25ce9753e (DEFAULT_TABLE table=default.Table_b32f665dddb043769e07076397f84a73 [id=f227576f105c4b1fbd459b3638ce684b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.566135 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13303d2a14c4187b82da9f25ce9753e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.529616 9341 catalog_manager.cc:5671] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:17.570012 9341 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]). Expected version 1 got 0
I20260504 13:56:17.571800 9341 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 48b193e958b4414aad29dd2bcee1a61b (table default.Table_2c39b14515bb46f6a0b0f1252d8e273d [id=e9e56499fd2243919308927b3614ae8e]). Expected version 1 got 0
I20260504 13:56:17.573207 9335 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 44017e2079be49258bab49325e3da1c1
I20260504 13:56:17.574746 9335 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35768 (request call id 138) took 2276 ms. Trace:
I20260504 13:56:17.540640 10003 raft_consensus.cc:2243] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:17.576645 10003 raft_consensus.cc:2272] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:17.579686 10003 ts_tablet_manager.cc:1916] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:17.584163 9814 tablet_service.cc:1558] Processing DeleteTablet for tablet b3793fe8f1ec43ad809403c4593f5ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:17.586149 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 91371bf848f5465ea2db0c090583382d (table default.Table_28052c777ee944b79b802d28b415c24e [id=994b65b95f66440283f852cd9dd9ae0c]) successfully deleted
W20260504 13:56:17.586544 9337 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 26) took 2882 ms (client timeout 1000 ms). Trace:
W20260504 13:56:17.587517 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f because the tablet was not found. No further retry: Not found: Tablet not found: bd8cd1caaeb249e3bafee7f1f0d4df2f
W20260504 13:56:17.588815 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet c34877a81d504f46a2f680c5d5ddb4f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c34877a81d504f46a2f680c5d5ddb4f4 already in progress: creating tablet
I20260504 13:56:17.575071 9335 rpcz_store.cc:276] 0504 13:56:15.298255 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:15.298410 (+ 155us) service_pool.cc:224] Handling call
0504 13:56:15.339818 (+ 41408us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:15.373142 (+ 33324us) catalog_manager.cc:2815] Modifying in-memory table state
0504 13:56:15.373281 (+ 139us) catalog_manager.cc:2824] Locking tablets
0504 13:56:16.243811 (+870530us) catalog_manager.cc:2838] Removing table and tablets from system table
0504 13:56:16.410895 (+167084us) catalog_manager.cc:2856] Removing table from by-name map
0504 13:56:16.411123 (+ 228us) catalog_manager.cc:2872] Committing in-memory state
0504 13:56:17.573114 (+1161991us) catalog_manager.cc:2876] Aborting table tasks
0504 13:56:17.574690 (+ 1576us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:16.375149 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:16.375533 (+ 384us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:16.375566 (+ 33us) write_op.cc:435] Acquired schema lock
0504 13:56:16.375578 (+ 12us) tablet.cc:662] Decoding operations
0504 13:56:16.375855 (+ 277us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:16.375886 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:16.375906 (+ 20us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:16.383270 (+ 7364us) tablet.cc:701] Row locks acquired
0504 13:56:16.383286 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:16.383387 (+ 101us) write_op.cc:270] Start()
0504 13:56:16.383459 (+ 72us) write_op.cc:276] Timestamp: P: 1777902976383334 usec, L: 0
0504 13:56:16.383474 (+ 15us) op_driver.cc:348] REPLICATION: starting
0504 13:56:16.383907 (+ 433us) log.cc:844] Serialized 771 byte log entry
0504 13:56:16.407773 (+ 23866us) op_driver.cc:464] REPLICATION: finished
0504 13:56:16.409256 (+ 1483us) write_op.cc:301] APPLY: starting
0504 13:56:16.409330 (+ 74us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:16.409463 (+ 133us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:16.409586 (+ 123us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:16.409859 (+ 273us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:16.410042 (+ 183us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:16.410115 (+ 73us) write_op.cc:312] APPLY: finished
0504 13:56:16.410351 (+ 236us) log.cc:844] Serialized 29 byte log entry
0504 13:56:16.410492 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:16.410559 (+ 67us) write_op.cc:454] Released schema lock
0504 13:56:16.410588 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:16.412025 (+ 1437us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:16.412063 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1265,"apply.run_cpu_time_us":1579,"apply.run_wall_time_us":2715,"num_ops":2,"prepare.queue_time_us":129529,"prepare.run_cpu_time_us":1965,"prepare.run_wall_time_us":16622,"raft.queue_time_us":209,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":11,"replication_time_us":24181}]]}
I20260504 13:56:17.592855 9335 master_service.cc:954] ReplaceTablet: received request to replace tablet b3793fe8f1ec43ad809403c4593f5ef6 from {username='slave'} at 127.0.0.1:35768
W20260504 13:56:17.591862 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f because the tablet was not found. No further retry: Not found: Tablet not found: bd8cd1caaeb249e3bafee7f1f0d4df2f
I20260504 13:56:17.593942 9814 tablet_service.cc:1511] Processing CreateTablet for tablet ab4c185332a24d96bc3d0fddc0d20693 (DEFAULT_TABLE table=default.Table_af058f50341e4f478c970f8cb52bf0d7 [id=57ccd7218d454c8c908fe06b798bd83b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.595531 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab4c185332a24d96bc3d0fddc0d20693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.597884 9343 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 3c43ff1d759a40d68344f25f066416ac
I20260504 13:56:17.601527 9932 tablet_bootstrap.cc:492] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:17.601923 9932 ts_tablet_manager.cc:1403] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.088s user 0.008s sys 0.000s
W20260504 13:56:17.586874 9337 rpcz_store.cc:269] 0504 13:56:14.704688 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:14.733138 (+ 28450us) service_pool.cc:224] Handling call
0504 13:56:17.586477 (+2853339us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:15.763019 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:15.763421 (+ 402us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:15.763443 (+ 22us) write_op.cc:435] Acquired schema lock
0504 13:56:15.763455 (+ 12us) tablet.cc:662] Decoding operations
0504 13:56:15.763851 (+ 396us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:15.763891 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:15.763912 (+ 21us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:15.764043 (+ 131us) tablet.cc:701] Row locks acquired
0504 13:56:15.764055 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:15.764159 (+ 104us) write_op.cc:270] Start()
0504 13:56:15.764232 (+ 73us) write_op.cc:276] Timestamp: P: 1777902975764112 usec, L: 0
0504 13:56:15.764246 (+ 14us) op_driver.cc:348] REPLICATION: starting
0504 13:56:15.764693 (+ 447us) log.cc:844] Serialized 533 byte log entry
0504 13:56:15.924529 (+159836us) op_driver.cc:464] REPLICATION: finished
0504 13:56:15.926448 (+ 1919us) write_op.cc:301] APPLY: starting
0504 13:56:15.926507 (+ 59us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:15.926594 (+ 87us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:15.926678 (+ 84us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:15.926828 (+ 150us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:15.926963 (+ 135us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:15.927007 (+ 44us) write_op.cc:312] APPLY: finished
0504 13:56:15.927214 (+ 207us) log.cc:844] Serialized 23 byte log entry
0504 13:56:15.927360 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:15.927418 (+ 58us) write_op.cc:454] Released schema lock
0504 13:56:15.927447 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:15.927703 (+ 256us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:15.927728 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":1762,"apply.run_cpu_time_us":1222,"apply.run_wall_time_us":1221,"num_ops":1,"prepare.queue_time_us":392756,"prepare.run_cpu_time_us":1812,"prepare.run_wall_time_us":4578,"raft.queue_time_us":76,"raft.run_cpu_time_us":575,"raft.run_wall_time_us":575,"replication_time_us":160186}]]}
I20260504 13:56:17.609932 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a43cdb767fd14ea09b4db4d2881b69aa" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:17.609826 9932 raft_consensus.cc:359] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.610479 9932 raft_consensus.cc:385] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:17.610528 9684 raft_consensus.cc:2468] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:17.610778 9932 raft_consensus.cc:740] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:17.611377 9932 consensus_queue.cc:260] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.563668 10013 raft_consensus.cc:359] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.616493 10013 raft_consensus.cc:385] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:17.616768 10013 raft_consensus.cc:740] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:17.617355 10013 consensus_queue.cc:260] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.617784 10273 consensus_queue.cc:1048] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:17.628849 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab4c185332a24d96bc3d0fddc0d20693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.628960 10002 ts_tablet_manager.cc:1929] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:17.629467 10002 log.cc:1199] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/0b7a71dbcc694e1a97019188cfb2c672
I20260504 13:56:17.630506 10002 ts_tablet_manager.cc:1950] T 0b7a71dbcc694e1a97019188cfb2c672 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:17.631493 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13303d2a14c4187b82da9f25ce9753e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.635552 9685 raft_consensus.cc:2468] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:17.636322 9932 ts_tablet_manager.cc:1434] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.034s user 0.004s sys 0.003s
I20260504 13:56:17.646107 9473 leader_election.cc:304] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:17.651736 10200 raft_consensus.cc:2804] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:17.657006 9932 tablet_bootstrap.cc:492] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:17.662914 9665 tablet_service.cc:1511] Processing CreateTablet for tablet b13303d2a14c4187b82da9f25ce9753e (DEFAULT_TABLE table=default.Table_b32f665dddb043769e07076397f84a73 [id=f227576f105c4b1fbd459b3638ce684b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.664885 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13303d2a14c4187b82da9f25ce9753e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.668030 10273 raft_consensus.cc:493] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:17.668509 10273 raft_consensus.cc:515] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.670624 10273 leader_election.cc:290] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:17.671350 10200 raft_consensus.cc:493] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:17.672036 10200 raft_consensus.cc:3060] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:17.672119 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a428049fdb4a68aae729400e2d9674" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:17.659529 10013 ts_tablet_manager.cc:1434] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.174s user 0.007s sys 0.001s
W20260504 13:56:17.673552 9618 leader_election.cc:343] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:17.661299 9473 leader_election.cc:304] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:17.677886 10013 tablet_bootstrap.cc:492] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:17.681407 9932 tablet_bootstrap.cc:654] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:17.681844 10277 raft_consensus.cc:2804] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:17.682626 10277 raft_consensus.cc:493] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:17.683156 10277 raft_consensus.cc:3060] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:17.688087 9473 consensus_peers.cc:597] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5625163f6add4a20b1828905d90b5486. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:17.689836 10195 consensus_queue.cc:1048] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:17.693066 9932 tablet_bootstrap.cc:492] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:17.693560 9932 ts_tablet_manager.cc:1403] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.004s
I20260504 13:56:17.699074 9810 tablet_service.cc:1558] Processing DeleteTablet for tablet 3c43ff1d759a40d68344f25f066416ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:17 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:17.703094 9808 tablet_service.cc:1558] Processing DeleteTablet for tablet 44017e2079be49258bab49325e3da1c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:15 UTC) from {username='slave'} at 127.0.0.1:56380
W20260504 13:56:17.711268 9328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 151006us
I20260504 13:56:17.711953 9932 raft_consensus.cc:359] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.712610 9932 raft_consensus.cc:385] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:17.713644 9932 raft_consensus.cc:740] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:17.714234 9932 consensus_queue.cc:260] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.718559 9513 tablet_service.cc:1558] Processing DeleteTablet for tablet 3c43ff1d759a40d68344f25f066416ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:17 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:17.719220 9514 tablet_service.cc:1558] Processing DeleteTablet for tablet 44017e2079be49258bab49325e3da1c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:15 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:17.719264 10195 consensus_queue.cc:1048] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:17.724573 9340 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 359b0fa01a1648a7bef6f97960a7b9a8 has no consensus state
I20260504 13:56:17.725260 9340 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35768 (request call id 144) took 2079 ms. Trace:
I20260504 13:56:17.729178 9932 ts_tablet_manager.cc:1434] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20260504 13:56:17.729709 10273 raft_consensus.cc:493] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:17.730127 10273 raft_consensus.cc:515] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.731981 10273 leader_election.cc:290] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:17.733397 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65052a5c28c147a385144317e21eea41" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:17.734735 9618 leader_election.cc:343] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:17.738546 9829 raft_consensus.cc:2468] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:17.739892 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c43ff1d759a40d68344f25f066416ac" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:17.744938 9932 tablet_bootstrap.cc:492] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:17.725555 9340 rpcz_store.cc:276] 0504 13:56:15.645804 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:16.039085 (+393281us) service_pool.cc:224] Handling call
0504 13:56:16.046880 (+ 7795us) spinlock_profiling.cc:243] Waited 6.47 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b0fb6 00007fc27b2c6b24 00007fc27b2c6ab2 00007fc27b2c69e1 00007fc27b2c695c 00007fc27b2c5c12 00007fc27958bbcc 00007fc27958b006
0504 13:56:16.047750 (+ 870us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:16.048549 (+ 799us) catalog_manager.cc:3725] Apply alter schema
0504 13:56:16.049237 (+ 688us) catalog_manager.cc:3757] Acquired catalog manager lock
0504 13:56:16.049414 (+ 177us) catalog_manager.cc:3915] Updating metadata on disk
0504 13:56:16.309730 (+260316us) catalog_manager.cc:3946] Committing alterations to in-memory state
0504 13:56:17.725175 (+1415445us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:16.253349 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:16.253739 (+ 390us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:16.253767 (+ 28us) write_op.cc:435] Acquired schema lock
0504 13:56:16.253783 (+ 16us) tablet.cc:662] Decoding operations
0504 13:56:16.254007 (+ 224us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:16.254043 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:16.254063 (+ 20us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:16.254208 (+ 145us) tablet.cc:701] Row locks acquired
0504 13:56:16.254220 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:16.254313 (+ 93us) write_op.cc:270] Start()
0504 13:56:16.254382 (+ 69us) write_op.cc:276] Timestamp: P: 1777902976254271 usec, L: 0
0504 13:56:16.254398 (+ 16us) op_driver.cc:348] REPLICATION: starting
0504 13:56:16.254789 (+ 391us) log.cc:844] Serialized 405 byte log entry
0504 13:56:16.307444 (+ 52655us) op_driver.cc:464] REPLICATION: finished
0504 13:56:16.307759 (+ 315us) write_op.cc:301] APPLY: starting
0504 13:56:16.307808 (+ 49us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:16.307896 (+ 88us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:16.307988 (+ 92us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:16.308134 (+ 146us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:16.308266 (+ 132us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:16.308332 (+ 66us) write_op.cc:312] APPLY: finished
0504 13:56:16.308538 (+ 206us) log.cc:844] Serialized 23 byte log entry
0504 13:56:16.308721 (+ 183us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:16.308799 (+ 78us) write_op.cc:454] Released schema lock
0504 13:56:16.308838 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:16.338387 (+ 29549us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:16.338412 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":14891776,"child_traces":[["op",{"apply.queue_time_us":178,"apply.run_cpu_time_us":1312,"apply.run_wall_time_us":1313,"num_ops":1,"prepare.queue_time_us":200951,"prepare.run_cpu_time_us":1945,"prepare.run_wall_time_us":85000,"raft.queue_time_us":11195,"raft.run_cpu_time_us":814,"raft.run_wall_time_us":816,"replication_time_us":52957}]]}
I20260504 13:56:17.752313 9932 tablet_bootstrap.cc:654] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:17.750167 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 73) took 2745 ms. Trace:
I20260504 13:56:17.753008 10002 rpcz_store.cc:276] 0504 13:56:15.005118 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:15.005246 (+ 128us) service_pool.cc:224] Handling call
0504 13:56:17.226661 (+2221415us) ts_tablet_manager.cc:1151] Deleting tablet 0b7a71dbcc694e1a97019188cfb2c672
0504 13:56:17.226699 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:17.439014 (+212315us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:17.628780 (+189766us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:17.750105 (+121325us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2220834}
I20260504 13:56:17.756096 10002 tablet_replica.cc:333] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:17.764604 9808 tablet_service.cc:1511] Processing CreateTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f (DEFAULT_TABLE table=default.Table_47c9ed0b601146b687a576ec89c8b626 [id=b75b9e5edced4443944af4087b77e54d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.764786 9810 tablet_service.cc:1511] Processing CreateTablet for tablet c34630d55f4a4e19b33ddf4c13995a5b (DEFAULT_TABLE table=default.Table_434bdc9867ad475a9719ba0c3760ef71 [id=db5cbb438d8f4da981e70b12713134f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.771983 10002 raft_consensus.cc:2243] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:17.772441 10002 raft_consensus.cc:2272] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:17.771227 9330 catalog_manager.cc:5075] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): alter failed for tablet 39f080a0296b44988e8e51e68453a551 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]),no further retry: Illegal state: Tablet not RUNNING: STOPPED
I20260504 13:56:17.774149 10002 ts_tablet_manager.cc:1916] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:17.774969 9932 tablet_bootstrap.cc:492] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:17.775429 9932 ts_tablet_manager.cc:1403] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.007s
I20260504 13:56:17.777639 9932 raft_consensus.cc:359] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.778373 9932 raft_consensus.cc:385] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:17.778649 9932 raft_consensus.cc:740] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:17.779213 9932 consensus_queue.cc:260] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.690070 9473 leader_election.cc:304] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:17.781075 9932 ts_tablet_manager.cc:1434] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260504 13:56:17.781939 9932 tablet_bootstrap.cc:492] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:17.784590 10195 raft_consensus.cc:2804] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:17.796725 9932 tablet_bootstrap.cc:654] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:17.800195 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 110245us
I20260504 13:56:17.806389 10273 raft_consensus.cc:493] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:17.806843 10273 raft_consensus.cc:515] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.808595 10273 leader_election.cc:290] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:17.808907 9931 tablet_bootstrap.cc:654] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:17.831310 9808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.833479 9810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34630d55f4a4e19b33ddf4c13995a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.856241 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 0b7a71dbcc694e1a97019188cfb2c672 (table default.Table_6aecd26cb4c04baea5252f5dfbb75663 [id=0649fc4627314bca8b51d0e1f5022e6d]) successfully deleted
I20260504 13:56:17.864900 9663 tablet_service.cc:1511] Processing CreateTablet for tablet ab4c185332a24d96bc3d0fddc0d20693 (DEFAULT_TABLE table=default.Table_af058f50341e4f478c970f8cb52bf0d7 [id=57ccd7218d454c8c908fe06b798bd83b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.867276 9660 tablet_service.cc:1558] Processing DeleteTablet for tablet 44017e2079be49258bab49325e3da1c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:15 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:17.869706 9660 tablet_service.cc:1558] Processing DeleteTablet for tablet 03818e34ac0545be880c873cf82cd4ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:11 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:17.870291 9664 tablet_service.cc:1558] Processing DeleteTablet for tablet b4dca0cadd5e400eab1365c03a8de4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 604f1ba5f71041a7877daff8972a994a) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:17.870301 9661 tablet_service.cc:1558] Processing DeleteTablet for tablet 0867df9ea115418193b6d9d3fc251923 with delete_type TABLET_DATA_DELETED (replaced by tablet c34877a81d504f46a2f680c5d5ddb4f4) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:17.871519 9664 tablet_service.cc:1558] Processing DeleteTablet for tablet 3c43ff1d759a40d68344f25f066416ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:17 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:17.805639 10284 consensus_queue.cc:1048] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:17.810035 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdb86f6370bf4c5f9c488d85efb67ef7" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:17.848531 9538 raft_consensus.cc:3060] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:17.872704 9661 tablet_service.cc:1511] Processing CreateTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f (DEFAULT_TABLE table=default.Table_47c9ed0b601146b687a576ec89c8b626 [id=b75b9e5edced4443944af4087b77e54d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.855780 9516 tablet_service.cc:1511] Processing CreateTablet for tablet 030b35c3409143108c34ab1f0ee1b96d (DEFAULT_TABLE table=default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.873847 9666 tablet_service.cc:1511] Processing CreateTablet for tablet c34630d55f4a4e19b33ddf4c13995a5b (DEFAULT_TABLE table=default.Table_434bdc9867ad475a9719ba0c3760ef71 [id=db5cbb438d8f4da981e70b12713134f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.874607 9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030b35c3409143108c34ab1f0ee1b96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.855997 9514 tablet_service.cc:1511] Processing CreateTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f (DEFAULT_TABLE table=default.Table_47c9ed0b601146b687a576ec89c8b626 [id=b75b9e5edced4443944af4087b77e54d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.879805 9514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.879228 10284 consensus_queue.cc:1048] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:17.881690 9338 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet a43cdb767fd14ea09b4db4d2881b69aa
I20260504 13:56:17.882907 9338 catalog_manager.cc:6741] ReplaceTablet: tablet a43cdb767fd14ea09b4db4d2881b69aa deleted and replaced by tablet 7c9030802900499a90c4dbaf5c6936a4
I20260504 13:56:17.857120 9513 tablet_service.cc:1511] Processing CreateTablet for tablet c34630d55f4a4e19b33ddf4c13995a5b (DEFAULT_TABLE table=default.Table_434bdc9867ad475a9719ba0c3760ef71 [id=db5cbb438d8f4da981e70b12713134f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.883262 10195 raft_consensus.cc:697] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:17.883790 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34630d55f4a4e19b33ddf4c13995a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.883814 9809 tablet_service.cc:1558] Processing DeleteTablet for tablet a43cdb767fd14ea09b4db4d2881b69aa with delete_type TABLET_DATA_DELETED (replaced by tablet 7c9030802900499a90c4dbaf5c6936a4) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:17.884042 10195 consensus_queue.cc:237] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.887939 9660 tablet_service.cc:1558] Processing DeleteTablet for tablet 0b7a71dbcc694e1a97019188cfb2c672 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:17.888566 9618 leader_election.cc:343] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:17.889703 9339 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:38691 (request call id 32) took 2262 ms (client timeout 1000 ms). Trace:
W20260504 13:56:17.890178 9339 rpcz_store.cc:269] 0504 13:56:15.626815 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:15.626968 (+ 153us) service_pool.cc:224] Handling call
0504 13:56:17.889522 (+2262554us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:16.228464 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:16.228836 (+ 372us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:16.228859 (+ 23us) write_op.cc:435] Acquired schema lock
0504 13:56:16.228870 (+ 11us) tablet.cc:662] Decoding operations
0504 13:56:16.229102 (+ 232us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:16.229149 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:16.229170 (+ 21us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:16.229298 (+ 128us) tablet.cc:701] Row locks acquired
0504 13:56:16.229310 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:16.229408 (+ 98us) write_op.cc:270] Start()
0504 13:56:16.229482 (+ 74us) write_op.cc:276] Timestamp: P: 1777902976229365 usec, L: 0
0504 13:56:16.229496 (+ 14us) op_driver.cc:348] REPLICATION: starting
0504 13:56:16.229933 (+ 437us) log.cc:844] Serialized 533 byte log entry
0504 13:56:16.239422 (+ 9489us) op_driver.cc:464] REPLICATION: finished
0504 13:56:16.239767 (+ 345us) write_op.cc:301] APPLY: starting
0504 13:56:16.239822 (+ 55us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:16.239922 (+ 100us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:16.240004 (+ 82us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:16.240148 (+ 144us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:16.240272 (+ 124us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:16.240316 (+ 44us) write_op.cc:312] APPLY: finished
0504 13:56:16.240472 (+ 156us) log.cc:844] Serialized 23 byte log entry
0504 13:56:16.240595 (+ 123us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:16.240658 (+ 63us) write_op.cc:454] Released schema lock
0504 13:56:16.240689 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:16.253074 (+ 12385us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:16.253104 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":213,"apply.run_cpu_time_us":1094,"apply.run_wall_time_us":1093,"num_ops":1,"prepare.queue_time_us":268743,"prepare.run_cpu_time_us":1872,"prepare.run_wall_time_us":24598,"raft.queue_time_us":132,"raft.run_cpu_time_us":623,"raft.run_wall_time_us":17723,"replication_time_us":9847}]]}
I20260504 13:56:17.894851 10284 raft_consensus.cc:493] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:17.895335 10284 raft_consensus.cc:515] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.897233 10284 leader_election.cc:290] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
W20260504 13:56:17.896119 9342 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:38691 (request call id 33) took 1286 ms (client timeout 1000 ms). Trace:
W20260504 13:56:17.903363 9342 rpcz_store.cc:269] 0504 13:56:16.609641 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:16.609762 (+ 121us) service_pool.cc:224] Handling call
0504 13:56:17.895938 (+1286176us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:17.310310 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:17.310812 (+ 502us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:17.310842 (+ 30us) write_op.cc:435] Acquired schema lock
0504 13:56:17.310861 (+ 19us) tablet.cc:662] Decoding operations
0504 13:56:17.311187 (+ 326us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:17.311233 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:17.311265 (+ 32us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:17.311432 (+ 167us) tablet.cc:701] Row locks acquired
0504 13:56:17.311448 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:17.311566 (+ 118us) write_op.cc:270] Start()
0504 13:56:17.311676 (+ 110us) write_op.cc:276] Timestamp: P: 1777902977311511 usec, L: 0
0504 13:56:17.311695 (+ 19us) op_driver.cc:348] REPLICATION: starting
0504 13:56:17.312222 (+ 527us) log.cc:844] Serialized 533 byte log entry
0504 13:56:17.467606 (+155384us) op_driver.cc:464] REPLICATION: finished
0504 13:56:17.468275 (+ 669us) write_op.cc:301] APPLY: starting
0504 13:56:17.468334 (+ 59us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:17.468427 (+ 93us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:17.468506 (+ 79us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:17.468658 (+ 152us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:17.468771 (+ 113us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:17.468814 (+ 43us) write_op.cc:312] APPLY: finished
0504 13:56:17.468974 (+ 160us) log.cc:844] Serialized 23 byte log entry
0504 13:56:17.469112 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:17.469162 (+ 50us) write_op.cc:454] Released schema lock
0504 13:56:17.469190 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:17.469398 (+ 208us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:17.469421 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":527,"apply.run_cpu_time_us":1090,"apply.run_wall_time_us":1091,"num_ops":1,"prepare.queue_time_us":252629,"prepare.run_cpu_time_us":2143,"prepare.run_wall_time_us":3277,"replication_time_us":155827}]]}
W20260504 13:56:17.909885 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 81094.3us
I20260504 13:56:17.915007 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7ab3b577b74a9c9a40cb15c94f8aa1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:17.920969 9474 leader_election.cc:343] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:17.928202 9660 tablet_service.cc:1511] Processing CreateTablet for tablet 030b35c3409143108c34ab1f0ee1b96d (DEFAULT_TABLE table=default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.928543 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a428049fdb4a68aae729400e2d9674" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:17.930158 9661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.932644 10273 consensus_queue.cc:1048] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:17.930724 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34630d55f4a4e19b33ddf4c13995a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.936906 9519 tablet_service.cc:1558] Processing DeleteTablet for tablet 14b7f1e834d0439c89abb180bd13853d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:17.937404 9520 tablet_service.cc:1558] Processing DeleteTablet for tablet 0867df9ea115418193b6d9d3fc251923 with delete_type TABLET_DATA_DELETED (replaced by tablet c34877a81d504f46a2f680c5d5ddb4f4) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:17.937531 9518 tablet_service.cc:1558] Processing DeleteTablet for tablet b3793fe8f1ec43ad809403c4593f5ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:17.939437 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030b35c3409143108c34ab1f0ee1b96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.940632 9809 tablet_service.cc:1511] Processing CreateTablet for tablet afad9329cf7745ba8168b2a93f998295 (DEFAULT_TABLE table=default.Table_cdee25533fe54da0a74b421714b0d7c4 [id=81df524d85774544807ba1902be1aae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.924690 9340 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
W20260504 13:56:17.936275 9328 consensus_peers.cc:597] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 -> Peer 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337): Couldn't send request to peer 6caedf407d13459e8fd477e8b60b7dc7. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260504 13:56:17.951458 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab4c185332a24d96bc3d0fddc0d20693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:17.974056 9932 tablet_bootstrap.cc:492] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:17.974840 9932 ts_tablet_manager.cc:1403] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.193s user 0.008s sys 0.017s
I20260504 13:56:17.976856 9338 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_15bf9089e68d4882838fe4f1c65156ef" } new_table_name: "default.Table_982e9b0871ac49a29936097dfa915691" modify_external_catalogs: true
I20260504 13:56:17.977473 9932 raft_consensus.cc:359] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.978216 9932 raft_consensus.cc:385] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:17.978506 9932 raft_consensus.cc:740] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:17.982512 9808 tablet_service.cc:1511] Processing CreateTablet for tablet 50e129980cd84f6ca22cbf2addb06230 (DEFAULT_TABLE table=default.Table_2717e0b44dd14ed1b25e45e06bc0ae88 [id=3e8d8cb6a0b049f8843286a4421e121a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:17.991395 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65052a5c28c147a385144317e21eea41" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:17.994469 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdb86f6370bf4c5f9c488d85efb67ef7" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:17.996959 9338 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 604f1ba5f71041a7877daff8972a994a consensus state has no leader
I20260504 13:56:18.003805 9809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afad9329cf7745ba8168b2a93f998295. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.005524 9808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e129980cd84f6ca22cbf2addb06230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.007798 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_5636294717ff4c63939c2cf9facfc788"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:18.009338 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5636294717ff4c63939c2cf9facfc788 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:17.951743 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_798d2540b8f4489fb11763ccaa36bced"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:18.012269 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_798d2540b8f4489fb11763ccaa36bced in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:18.029774 9335 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_06ecf48a04d94ac8a7e954ff2c260dbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:18.031833 9335 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06ecf48a04d94ac8a7e954ff2c260dbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:18.036713 9343 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_6546f8c7ece54e92b8f96bd957ae5851"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:18.038348 9343 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6546f8c7ece54e92b8f96bd957ae5851 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:18.039062 9813 tablet_service.cc:1511] Processing CreateTablet for tablet 030b35c3409143108c34ab1f0ee1b96d (DEFAULT_TABLE table=default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.036648 9337 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_c6a898ba23b74b54bb4684e970b44742" } modify_external_catalogs: true
I20260504 13:56:18.039772 9337 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_c6a898ba23b74b54bb4684e970b44742" } modify_external_catalogs: true
I20260504 13:56:18.040040 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030b35c3409143108c34ab1f0ee1b96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.044234 10215 raft_consensus.cc:493] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:18.044684 10215 raft_consensus.cc:515] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.045261 9931 tablet_bootstrap.cc:492] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:18.045630 9931 ts_tablet_manager.cc:1403] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.523s user 0.012s sys 0.005s
I20260504 13:56:18.046337 10215 leader_election.cc:290] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
W20260504 13:56:17.970945 9336 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 27) took 2181 ms (client timeout 1000 ms). Trace:
I20260504 13:56:18.049916 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70caac2e256d419dbc2e70cb5dfffe97" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:18.047518 9336 rpcz_store.cc:269] 0504 13:56:15.789498 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:15.934814 (+145316us) service_pool.cc:224] Handling call
0504 13:56:17.970766 (+2035952us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:16.973970 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:16.974325 (+ 355us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:16.974346 (+ 21us) write_op.cc:435] Acquired schema lock
0504 13:56:16.974358 (+ 12us) tablet.cc:662] Decoding operations
0504 13:56:16.974571 (+ 213us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:16.974605 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:16.974626 (+ 21us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:16.974743 (+ 117us) tablet.cc:701] Row locks acquired
0504 13:56:16.974755 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:16.974843 (+ 88us) write_op.cc:270] Start()
0504 13:56:16.974930 (+ 87us) write_op.cc:276] Timestamp: P: 1777902976974796 usec, L: 0
0504 13:56:16.974951 (+ 21us) op_driver.cc:348] REPLICATION: starting
0504 13:56:16.975495 (+ 544us) log.cc:844] Serialized 533 byte log entry
0504 13:56:16.985004 (+ 9509us) op_driver.cc:464] REPLICATION: finished
0504 13:56:16.986423 (+ 1419us) write_op.cc:301] APPLY: starting
0504 13:56:16.986472 (+ 49us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:16.986559 (+ 87us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:16.986648 (+ 89us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:16.986938 (+ 290us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:16.987097 (+ 159us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:16.987157 (+ 60us) write_op.cc:312] APPLY: finished
0504 13:56:16.987388 (+ 231us) log.cc:844] Serialized 23 byte log entry
0504 13:56:16.987541 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:16.987609 (+ 68us) write_op.cc:454] Released schema lock
0504 13:56:16.987650 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:16.995826 (+ 8176us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:16.995859 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5592,"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":1275,"apply.run_cpu_time_us":1340,"apply.run_wall_time_us":1425,"num_ops":1,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":1876,"prepare.run_wall_time_us":21822,"raft.queue_time_us":1078,"raft.run_cpu_time_us":645,"raft.run_wall_time_us":969,"replication_time_us":9963}]]}
W20260504 13:56:18.052058 9618 leader_election.cc:343] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:18.054589 9518 tablet_service.cc:1558] Processing DeleteTablet for tablet a43cdb767fd14ea09b4db4d2881b69aa with delete_type TABLET_DATA_DELETED (replaced by tablet 7c9030802900499a90c4dbaf5c6936a4) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:18.056592 9931 raft_consensus.cc:359] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.057175 9931 raft_consensus.cc:385] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:18.057443 9931 raft_consensus.cc:740] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:18.063675 10013 tablet_bootstrap.cc:700] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent opening tablet: real 0.381s user 0.003s sys 0.000s
I20260504 13:56:18.074364 10200 raft_consensus.cc:515] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:17.979059 9932 consensus_queue.cc:260] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.076152 10200 leader_election.cc:290] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:18.077356 9932 ts_tablet_manager.cc:1434] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.102s user 0.005s sys 0.000s
I20260504 13:56:18.077692 9518 tablet_service.cc:1511] Processing CreateTablet for tablet afad9329cf7745ba8168b2a93f998295 (DEFAULT_TABLE table=default.Table_cdee25533fe54da0a74b421714b0d7c4 [id=81df524d85774544807ba1902be1aae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.078707 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afad9329cf7745ba8168b2a93f998295. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.080580 10013 tablet_bootstrap.cc:654] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:18.100039 9931 consensus_queue.cc:260] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.101469 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7ab3b577b74a9c9a40cb15c94f8aa1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:18.102394 9686 raft_consensus.cc:2468] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:18.105019 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "baf07a970a7c4a35b3c849b06d6c88d0" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:18.106396 9474 leader_election.cc:343] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:18.107045 10277 raft_consensus.cc:515] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.115283 9931 ts_tablet_manager.cc:1434] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.069s user 0.007s sys 0.000s
I20260504 13:56:18.117828 9944 ts_tablet_manager.cc:1929] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20260504 13:56:18.118177 9944 log.cc:1199] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/39f080a0296b44988e8e51e68453a551
I20260504 13:56:18.118430 9931 tablet_bootstrap.cc:492] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:18.118840 9944 ts_tablet_manager.cc:1950] T 39f080a0296b44988e8e51e68453a551 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:18.124742 10003 ts_tablet_manager.cc:1929] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20260504 13:56:18.145373 9932 tablet_bootstrap.cc:492] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:18.149441 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "baf07a970a7c4a35b3c849b06d6c88d0" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:18.150235 9686 raft_consensus.cc:3060] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:18.127019 10002 ts_tablet_manager.cc:1929] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:18.155890 10002 log.cc:1199] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/d7e556f81ec940f1aa3c55b3a25f31ab
I20260504 13:56:18.157858 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a43cdb767fd14ea09b4db4d2881b69aa" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:18.158411 9683 raft_consensus.cc:3060] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:18.159909 10213 raft_consensus.cc:493] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:18.160055 10002 ts_tablet_manager.cc:1950] T d7e556f81ec940f1aa3c55b3a25f31ab P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:18.160383 10213 raft_consensus.cc:515] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:18.168093 9330 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet b8e4239942f34fb081a026a35740e465 (table default.Table_11d8ab61a6044154bb599875a0bc9004 [id=73e12524eb9e416782bf9f7fb58c93bd], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b8e4239942f34fb081a026a35740e465 consensus state has no leader
I20260504 13:56:18.169669 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 74) took 3163 ms. Trace:
I20260504 13:56:18.164671 10013 tablet_bootstrap.cc:492] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:18.170289 10013 ts_tablet_manager.cc:1403] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.493s user 0.011s sys 0.000s
I20260504 13:56:18.172555 10013 raft_consensus.cc:359] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.173198 10013 raft_consensus.cc:385] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:18.173499 10013 raft_consensus.cc:740] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:18.175186 10013 consensus_queue.cc:260] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.075841 9338 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_d5cc86d81fb043319576af0a39c643a8" } modify_external_catalogs: true
I20260504 13:56:18.178740 9338 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_d5cc86d81fb043319576af0a39c643a8" } modify_external_catalogs: true
I20260504 13:56:18.181627 10277 leader_election.cc:290] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:18.150532 9339 catalog_manager.cc:5671] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
W20260504 13:56:18.185089 9620 leader_election.cc:343] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:18.185493 9620 leader_election.cc:304] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:18.185325 9339 catalog_manager.cc:5671] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:18.187065 9339 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:18.198465 10273 raft_consensus.cc:493] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:18.198877 10273 raft_consensus.cc:515] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.206171 9515 tablet_service.cc:1511] Processing CreateTablet for tablet 50e129980cd84f6ca22cbf2addb06230 (DEFAULT_TABLE table=default.Table_2717e0b44dd14ed1b25e45e06bc0ae88 [id=3e8d8cb6a0b049f8843286a4421e121a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.207630 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e129980cd84f6ca22cbf2addb06230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.213831 10002 rpcz_store.cc:276] 0504 13:56:15.006235 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:15.006323 (+ 88us) service_pool.cc:224] Handling call
0504 13:56:17.755907 (+2749584us) ts_tablet_manager.cc:1151] Deleting tablet d7e556f81ec940f1aa3c55b3a25f31ab
0504 13:56:17.755960 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:17.946515 (+190555us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.126895 (+180380us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.169610 (+ 42715us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2749067}
I20260504 13:56:18.182822 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a43cdb767fd14ea09b4db4d2881b69aa" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:18.222483 10003 log.cc:1199] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/39f080a0296b44988e8e51e68453a551
I20260504 13:56:18.223270 10003 ts_tablet_manager.cc:1950] T 39f080a0296b44988e8e51e68453a551 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
W20260504 13:56:18.222421 9474 leader_election.cc:343] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:18.229475 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 78) took 2423 ms. Trace:
I20260504 13:56:18.236819 10002 rpcz_store.cc:276] 0504 13:56:15.806439 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:15.806543 (+ 104us) service_pool.cc:224] Handling call
0504 13:56:18.229111 (+2422568us) ts_tablet_manager.cc:1151] Deleting tablet 07a428049fdb4a68aae729400e2d9674
0504 13:56:18.229159 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:18.229414 (+ 255us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2422030}
I20260504 13:56:18.238173 10003 tablet_replica.cc:333] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:18.235801 10013 ts_tablet_manager.cc:1434] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.065s user 0.005s sys 0.000s
I20260504 13:56:18.251973 10013 tablet_bootstrap.cc:492] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:18.225677 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 39f080a0296b44988e8e51e68453a551 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]) successfully deleted
I20260504 13:56:18.258038 10002 tablet_replica.cc:333] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:18.260646 10002 raft_consensus.cc:2243] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:18.257151 9339 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 48b193e958b4414aad29dd2bcee1a61b (table default.Table_2c39b14515bb46f6a0b0f1252d8e273d [id=e9e56499fd2243919308927b3614ae8e]). Expected version 1 got 0
I20260504 13:56:18.263151 9339 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]). Expected version 1 got 0
I20260504 13:56:18.268127 10002 raft_consensus.cc:2272] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:18.226500 9341 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 28) took 1351 ms (client timeout 1000 ms). Trace:
I20260504 13:56:18.260619 9336 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_960fc67a53724e78bcb430e7fdef3151" } new_table_name: "default.Table_6a56efa14aa6497d9926d20508348656" modify_external_catalogs: true
W20260504 13:56:18.270577 9341 rpcz_store.cc:269] 0504 13:56:16.875060 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:16.883991 (+ 8931us) service_pool.cc:224] Handling call
0504 13:56:16.901698 (+ 17707us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:18.226334 (+1324636us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:17.820461 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:17.820915 (+ 454us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:17.820951 (+ 36us) write_op.cc:435] Acquired schema lock
0504 13:56:17.820963 (+ 12us) tablet.cc:662] Decoding operations
0504 13:56:17.821289 (+ 326us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:17.821345 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:17.821379 (+ 34us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:17.821586 (+ 207us) tablet.cc:701] Row locks acquired
0504 13:56:17.821600 (+ 14us) write_op.cc:260] PREPARE: finished
0504 13:56:17.821715 (+ 115us) write_op.cc:270] Start()
0504 13:56:17.821803 (+ 88us) write_op.cc:276] Timestamp: P: 1777902977821626 usec, L: 0
0504 13:56:17.821835 (+ 32us) op_driver.cc:348] REPLICATION: starting
0504 13:56:17.822568 (+ 733us) log.cc:844] Serialized 533 byte log entry
0504 13:56:17.830502 (+ 7934us) op_driver.cc:464] REPLICATION: finished
0504 13:56:17.831091 (+ 589us) write_op.cc:301] APPLY: starting
0504 13:56:17.831159 (+ 68us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:17.831302 (+ 143us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:17.831438 (+ 136us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:17.831690 (+ 252us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:17.831902 (+ 212us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:17.831976 (+ 74us) write_op.cc:312] APPLY: finished
0504 13:56:17.832218 (+ 242us) log.cc:844] Serialized 23 byte log entry
0504 13:56:17.832367 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:17.832440 (+ 73us) write_op.cc:454] Released schema lock
0504 13:56:17.832484 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:17.832725 (+ 241us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:17.832752 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":20,"spinlock_wait_cycles":30512896,"child_traces":[["op",{"apply.queue_time_us":354,"apply.run_cpu_time_us":1604,"apply.run_wall_time_us":1603,"num_ops":1,"prepare.queue_time_us":73360,"prepare.run_cpu_time_us":2507,"prepare.run_wall_time_us":2507,"raft.queue_time_us":1515,"raft.run_cpu_time_us":742,"raft.run_wall_time_us":743,"replication_time_us":8558}]]}
I20260504 13:56:18.289012 10002 ts_tablet_manager.cc:1916] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:18.292546 10215 raft_consensus.cc:2749] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:18.293331 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 108265us
W20260504 13:56:18.293959 9620 leader_election.cc:343] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:18.294411 9620 leader_election.cc:304] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:18.295003 10215 raft_consensus.cc:2749] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:18.295185 10273 leader_election.cc:290] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:18.296078 10215 consensus_queue.cc:1048] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:18.297866 9620 leader_election.cc:343] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:18.298269 9620 leader_election.cc:304] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:18.284034 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet d7e556f81ec940f1aa3c55b3a25f31ab (table default.Table_ea18464634544273a0bbb10fce8b88ba [id=2388ca57302443e48f66ac3a8db39000]) successfully deleted
W20260504 13:56:18.299629 9328 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 54159.6us
I20260504 13:56:18.299767 10003 raft_consensus.cc:2243] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:18.300231 10003 raft_consensus.cc:2272] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:18.280048 9808 tablet_service.cc:1511] Processing CreateTablet for tablet 604f1ba5f71041a7877daff8972a994a (DEFAULT_TABLE table=default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.301203 9808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 604f1ba5f71041a7877daff8972a994a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.287132 9813 tablet_service.cc:1511] Processing CreateTablet for tablet dd3c045379734134b49e94340679c11d (DEFAULT_TABLE table=default.Table_443402c40ae5443daf4ba4258b847303 [id=98126e5c27cd4d3da57fcd88d481de79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.302189 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd3c045379734134b49e94340679c11d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.188844 9931 tablet_bootstrap.cc:654] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:18.295631 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03fe4ce4f52f48b7b7a804c269e0fd4e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:18.311807 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 07a428049fdb4a68aae729400e2d9674 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a428049fdb4a68aae729400e2d9674 already in progress: creating tablet
I20260504 13:56:18.317297 10003 ts_tablet_manager.cc:1916] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:18.330742 9344 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:18.298920 10013 tablet_bootstrap.cc:654] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:18.299829 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1501455a0af348e38746d100987afdd0" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:18.301216 9517 tablet_service.cc:1511] Processing CreateTablet for tablet 604f1ba5f71041a7877daff8972a994a (DEFAULT_TABLE table=default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.332333 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 604f1ba5f71041a7877daff8972a994a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.332788 9344 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]). Expected version 1 got 0
I20260504 13:56:18.333348 9344 catalog_manager.cc:5671] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:18.334435 9344 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 48b193e958b4414aad29dd2bcee1a61b (table default.Table_2c39b14515bb46f6a0b0f1252d8e273d [id=e9e56499fd2243919308927b3614ae8e]). Expected version 1 got 0
I20260504 13:56:18.340786 9810 tablet_service.cc:1511] Processing CreateTablet for tablet 7ed3ee9a13e84594905db47aa33040c2 (DEFAULT_TABLE table=default.Table_6462cb42c34a4eb8976d8d2a4f82d434 [id=6e3151859b6d4da3a477367687862671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.342162 9810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ed3ee9a13e84594905db47aa33040c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.349545 9341 master_service.cc:954] ReplaceTablet: received request to replace tablet 44017e2079be49258bab49325e3da1c1 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:18.350756 9518 tablet_service.cc:1511] Processing CreateTablet for tablet dd3c045379734134b49e94340679c11d (DEFAULT_TABLE table=default.Table_443402c40ae5443daf4ba4258b847303 [id=98126e5c27cd4d3da57fcd88d481de79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.353350 9931 tablet_bootstrap.cc:492] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:18.353806 9931 ts_tablet_manager.cc:1403] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.236s user 0.011s sys 0.000s
I20260504 13:56:18.356166 9931 raft_consensus.cc:359] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.356835 9931 raft_consensus.cc:385] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:18.357132 9931 raft_consensus.cc:740] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:18.357788 9931 consensus_queue.cc:260] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.362334 9931 ts_tablet_manager.cc:1434] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
W20260504 13:56:18.364225 9336 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a1cf52a9e58744a99dae1958d2c1f867 consensus state has no leader
I20260504 13:56:18.367543 10215 raft_consensus.cc:2749] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:18.368242 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 70399.8us
W20260504 13:56:18.368865 9620 consensus_peers.cc:597] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:18.369321 9620 leader_election.cc:343] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:18.369648 9620 leader_election.cc:304] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:18.300817 10213 leader_election.cc:290] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:18.304672 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 83) took 1257 ms. Trace:
W20260504 13:56:18.307943 9618 leader_election.cc:343] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:18.370386 9618 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 62471.9us
W20260504 13:56:18.371263 9618 leader_election.cc:343] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:18.372193 9944 rpcz_store.cc:276] 0504 13:56:17.047105 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.047406 (+ 301us) service_pool.cc:224] Handling call
0504 13:56:17.437374 (+389968us) ts_tablet_manager.cc:1151] Deleting tablet 39f080a0296b44988e8e51e68453a551
0504 13:56:17.437408 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:17.979400 (+541992us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.117713 (+138313us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.304544 (+186831us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":389212}
I20260504 13:56:18.373497 9944 tablet_replica.cc:333] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:18.374248 9944 raft_consensus.cc:2243] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:18.381142 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd3c045379734134b49e94340679c11d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.335136 9344 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet afee197774db411d97468913a207703a (table default.Table_91c2d3ed5c5c42758c87f1f726af88c0 [id=a2697dba5f38435c81dc52316712917d]). Expected version 2 got 0
I20260504 13:56:18.384093 9683 raft_consensus.cc:2468] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:18.384792 9344 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 70caac2e256d419dbc2e70cb5dfffe97 (table default.Table_c5584202645f4ef78e4f6e00aa2bf2c1 [id=2665806fb81f4eda8981fede847cbfcf]). Expected version 1 got 0
I20260504 13:56:18.386873 9515 tablet_service.cc:1511] Processing CreateTablet for tablet 7ed3ee9a13e84594905db47aa33040c2 (DEFAULT_TABLE table=default.Table_6462cb42c34a4eb8976d8d2a4f82d434 [id=6e3151859b6d4da3a477367687862671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.387470 9336 master_service.cc:954] ReplaceTablet: received request to replace tablet b3793fe8f1ec43ad809403c4593f5ef6 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:18.387745 9338 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet eb7f58645c76441589368072274ebc8f
I20260504 13:56:18.315007 9932 tablet_bootstrap.cc:654] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:18.390360 10215 raft_consensus.cc:2749] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:18.391774 9340 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: default.Table_9e9ef3353dcf4b1fadaa2f51bc8f7041 [id=8dcabd7d67334598926f927a62e630d6] alter complete (version 2)
I20260504 13:56:18.395123 10213 consensus_queue.cc:1048] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:18.399502 9944 raft_consensus.cc:2272] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:18.401701 9944 ts_tablet_manager.cc:1916] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:18.407742 9337 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 359b0fa01a1648a7bef6f97960a7b9a8
I20260504 13:56:18.385864 9808 tablet_service.cc:1511] Processing CreateTablet for tablet a1cf52a9e58744a99dae1958d2c1f867 (DEFAULT_TABLE table=default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.409895 9808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1cf52a9e58744a99dae1958d2c1f867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.416244 9931 tablet_bootstrap.cc:492] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:18.390336 10003 ts_tablet_manager.cc:1929] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:18.391003 9809 tablet_service.cc:1558] Processing DeleteTablet for tablet eb7f58645c76441589368072274ebc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:18 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:18.394800 9812 tablet_service.cc:1558] Processing DeleteTablet for tablet 69ed77782cd34274932957106ce26766 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:18.418596 10003 log.cc:1199] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/b3793fe8f1ec43ad809403c4593f5ef6
I20260504 13:56:18.419243 10003 ts_tablet_manager.cc:1950] T b3793fe8f1ec43ad809403c4593f5ef6 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
W20260504 13:56:18.413584 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 44277.4us
I20260504 13:56:18.426947 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet b3793fe8f1ec43ad809403c4593f5ef6 (table default.Table_61add6c7b0dd46b08c0e39ad6d697e8c [id=1749c1a022544b1185fb3d4a4a530afd]) successfully deleted
W20260504 13:56:18.428398 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 3c43ff1d759a40d68344f25f066416ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c43ff1d759a40d68344f25f066416ac already in progress: creating tablet
I20260504 13:56:18.416553 9686 raft_consensus.cc:2468] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:18.441992 9663 tablet_service.cc:1558] Processing DeleteTablet for tablet 91371bf848f5465ea2db0c090583382d with delete_type TABLET_DATA_DELETED (replaced by tablet 60d3cc59ffe947fbb57b7d3301ccbac1) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:18.442229 9665 tablet_service.cc:1558] Processing DeleteTablet for tablet a43cdb767fd14ea09b4db4d2881b69aa with delete_type TABLET_DATA_DELETED (replaced by tablet 7c9030802900499a90c4dbaf5c6936a4) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:18.450618 10003 tablet_replica.cc:333] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:18.451313 10003 raft_consensus.cc:2243] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:18.451880 10003 raft_consensus.cc:2272] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:18.453848 10003 ts_tablet_manager.cc:1916] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:18.453904 9660 tablet_service.cc:1558] Processing DeleteTablet for tablet 14b7f1e834d0439c89abb180bd13853d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:18.387288 9513 tablet_service.cc:1511] Processing CreateTablet for tablet a1cf52a9e58744a99dae1958d2c1f867 (DEFAULT_TABLE table=default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.455426 9660 tablet_service.cc:1558] Processing DeleteTablet for tablet 69ed77782cd34274932957106ce26766 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:18.456249 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1cf52a9e58744a99dae1958d2c1f867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.458359 9660 tablet_service.cc:1558] Processing DeleteTablet for tablet b3793fe8f1ec43ad809403c4593f5ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:18.459408 9338 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_fa64d5b0eb084f2a9ab924aa2cdf3a1c" } new_table_name: "default.Table_0d4595642a604b8f8f57d3eadbbacacb" modify_external_catalogs: true
I20260504 13:56:18.418179 9517 tablet_service.cc:1558] Processing DeleteTablet for tablet eb7f58645c76441589368072274ebc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:18 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:18.432796 10013 tablet_bootstrap.cc:492] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:18.440771 9514 tablet_service.cc:1558] Processing DeleteTablet for tablet 0b7a71dbcc694e1a97019188cfb2c672 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:18.452637 9516 tablet_service.cc:1558] Processing DeleteTablet for tablet d7e556f81ec940f1aa3c55b3a25f31ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:18.462709 10013 ts_tablet_manager.cc:1403] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.211s user 0.010s sys 0.000s
I20260504 13:56:18.453356 9520 tablet_service.cc:1558] Processing DeleteTablet for tablet 69ed77782cd34274932957106ce26766 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:18.464995 10013 raft_consensus.cc:359] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.465628 10013 raft_consensus.cc:385] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:18.465859 10013 raft_consensus.cc:740] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:18.466480 10013 consensus_queue.cc:260] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.468483 9932 tablet_bootstrap.cc:492] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:18.469717 9932 ts_tablet_manager.cc:1403] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.325s user 0.012s sys 0.004s
I20260504 13:56:18.477325 9932 raft_consensus.cc:359] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.480628 10013 ts_tablet_manager.cc:1434] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20260504 13:56:18.482882 9932 raft_consensus.cc:385] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:18.483511 9932 raft_consensus.cc:740] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:18.459061 9931 tablet_bootstrap.cc:654] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:18.484601 9932 consensus_queue.cc:260] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.485332 9660 tablet_service.cc:1558] Processing DeleteTablet for tablet d7e556f81ec940f1aa3c55b3a25f31ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:18.487277 9663 tablet_service.cc:1511] Processing CreateTablet for tablet 50e129980cd84f6ca22cbf2addb06230 (DEFAULT_TABLE table=default.Table_2717e0b44dd14ed1b25e45e06bc0ae88 [id=3e8d8cb6a0b049f8843286a4421e121a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.491865 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e129980cd84f6ca22cbf2addb06230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.491802 9666 tablet_service.cc:1511] Processing CreateTablet for tablet 604f1ba5f71041a7877daff8972a994a (DEFAULT_TABLE table=default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.492678 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 604f1ba5f71041a7877daff8972a994a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.492878 10002 ts_tablet_manager.cc:1929] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:18.493081 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ed3ee9a13e84594905db47aa33040c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.493213 10002 log.cc:1199] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/5625163f6add4a20b1828905d90b5486
I20260504 13:56:18.493875 10002 ts_tablet_manager.cc:1950] T 5625163f6add4a20b1828905d90b5486 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:18.497515 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 81) took 2518 ms. Trace:
I20260504 13:56:18.497815 10002 rpcz_store.cc:276] 0504 13:56:16.015599 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:16.015726 (+ 127us) service_pool.cc:224] Handling call
0504 13:56:18.257905 (+2242179us) ts_tablet_manager.cc:1151] Deleting tablet 5625163f6add4a20b1828905d90b5486
0504 13:56:18.257944 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:18.389912 (+131968us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.492772 (+102860us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.497456 (+ 4684us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2241572}
I20260504 13:56:18.487460 9665 tablet_service.cc:1511] Processing CreateTablet for tablet afad9329cf7745ba8168b2a93f998295 (DEFAULT_TABLE table=default.Table_cdee25533fe54da0a74b421714b0d7c4 [id=81df524d85774544807ba1902be1aae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.503875 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afad9329cf7745ba8168b2a93f998295. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.504036 10013 tablet_bootstrap.cc:492] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:18.509682 10013 tablet_bootstrap.cc:654] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:18.520067 9343 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_7aba2a5065e04c10a7116b73aabdf238"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:18.521613 9343 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7aba2a5065e04c10a7116b73aabdf238 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:18.523761 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_a8bc83249ad949c49e759a07818e2c02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:18.525069 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8bc83249ad949c49e759a07818e2c02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:18.527359 10002 tablet_replica.cc:333] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:18.528069 10002 raft_consensus.cc:2243] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:18.528937 10002 raft_consensus.cc:2272] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:18.529163 9335 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_05a212a8397548bd94b840d6b4cf4fd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:18.530682 10002 ts_tablet_manager.cc:1916] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260504 13:56:18.530646 9335 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05a212a8397548bd94b840d6b4cf4fd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:18.531544 9932 ts_tablet_manager.cc:1434] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.056s user 0.003s sys 0.002s
I20260504 13:56:18.536666 9932 tablet_bootstrap.cc:492] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:18.537578 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 5625163f6add4a20b1828905d90b5486 (table default.Table_56891f8543a84e859c39b381625a9daa [id=7b3ef250728944d9998ef2df032f8819]) successfully deleted
I20260504 13:56:18.538763 9340 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:18.544551 9660 tablet_service.cc:1511] Processing CreateTablet for tablet 7ed3ee9a13e84594905db47aa33040c2 (DEFAULT_TABLE table=default.Table_6462cb42c34a4eb8976d8d2a4f82d434 [id=6e3151859b6d4da3a477367687862671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.545581 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ed3ee9a13e84594905db47aa33040c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.550024 9337 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:18.550810 9337 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 48b193e958b4414aad29dd2bcee1a61b (table default.Table_2c39b14515bb46f6a0b0f1252d8e273d [id=e9e56499fd2243919308927b3614ae8e]). Expected version 1 got 0
I20260504 13:56:18.558264 9663 tablet_service.cc:1511] Processing CreateTablet for tablet dd3c045379734134b49e94340679c11d (DEFAULT_TABLE table=default.Table_443402c40ae5443daf4ba4258b847303 [id=98126e5c27cd4d3da57fcd88d481de79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.559684 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd3c045379734134b49e94340679c11d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.561965 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_9bc232a8e4dd4df1a36b35d09dd50d0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:18.551041 9932 tablet_bootstrap.cc:654] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:18.563374 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bc232a8e4dd4df1a36b35d09dd50d0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:18.565201 9931 tablet_bootstrap.cc:492] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:18.565596 9931 ts_tablet_manager.cc:1403] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.150s user 0.009s sys 0.001s
I20260504 13:56:18.567993 9931 raft_consensus.cc:359] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.568593 9931 raft_consensus.cc:385] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:18.568840 9931 raft_consensus.cc:740] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:18.569621 9931 consensus_queue.cc:260] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.572466 9661 tablet_service.cc:1558] Processing DeleteTablet for tablet eb7f58645c76441589368072274ebc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:18 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:18.573609 9664 tablet_service.cc:1511] Processing CreateTablet for tablet a1cf52a9e58744a99dae1958d2c1f867 (DEFAULT_TABLE table=default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:18.575116 9664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1cf52a9e58744a99dae1958d2c1f867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:18.578538 9931 ts_tablet_manager.cc:1434] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260504 13:56:18.584789 9931 tablet_bootstrap.cc:492] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:18.591161 9931 tablet_bootstrap.cc:654] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:18.592232 10013 tablet_bootstrap.cc:492] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:18.592619 10013 ts_tablet_manager.cc:1403] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.005s
I20260504 13:56:18.607263 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70caac2e256d419dbc2e70cb5dfffe97" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:18.609529 10013 raft_consensus.cc:359] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.610123 10013 raft_consensus.cc:385] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:18.610378 10013 raft_consensus.cc:740] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:18.610898 10013 consensus_queue.cc:260] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.613459 10013 ts_tablet_manager.cc:1434] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.021s user 0.005s sys 0.002s
I20260504 13:56:18.617023 10013 tablet_bootstrap.cc:492] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:18.622677 9830 raft_consensus.cc:3060] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:18.623982 10277 consensus_queue.cc:1048] T 4fdc09a498be470ab04563c62caf7755 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:18.637740 9473 leader_election.cc:304] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:18.639027 10277 raft_consensus.cc:2804] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:18.639369 10277 raft_consensus.cc:493] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:18.639637 10277 raft_consensus.cc:3060] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:18.643132 9932 tablet_bootstrap.cc:492] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:18.643966 9932 ts_tablet_manager.cc:1403] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.112s user 0.008s sys 0.001s
I20260504 13:56:18.644294 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1501455a0af348e38746d100987afdd0" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:18.646416 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 39f080a0296b44988e8e51e68453a551 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]) successfully deleted
I20260504 13:56:18.646490 9932 raft_consensus.cc:359] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.648308 10002 ts_tablet_manager.cc:1929] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:18.648721 10002 log.cc:1199] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/91371bf848f5465ea2db0c090583382d
I20260504 13:56:18.649544 10002 ts_tablet_manager.cc:1950] T 91371bf848f5465ea2db0c090583382d P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:18.653198 10314 tablet.cc:1724] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:18.655933 10315 tablet.cc:1724] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:18.658447 10277 raft_consensus.cc:515] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.665370 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7ab3b577b74a9c9a40cb15c94f8aa1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:18.666761 9474 leader_election.cc:343] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:18.667323 10277 leader_election.cc:290] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:18.668231 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03fe4ce4f52f48b7b7a804c269e0fd4e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:18.668605 10003 ts_tablet_manager.cc:1929] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:18.669004 10003 log.cc:1199] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/44017e2079be49258bab49325e3da1c1
I20260504 13:56:18.669813 10003 ts_tablet_manager.cc:1950] T 44017e2079be49258bab49325e3da1c1 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:18.671617 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 84) took 2213 ms. Trace:
I20260504 13:56:18.671964 10002 rpcz_store.cc:276] 0504 13:56:16.457885 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:16.457995 (+ 110us) service_pool.cc:224] Handling call
0504 13:56:18.499275 (+2041280us) ts_tablet_manager.cc:1151] Deleting tablet 91371bf848f5465ea2db0c090583382d
0504 13:56:18.527227 (+ 27952us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:18.601447 (+ 74220us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.648197 (+ 46750us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.671557 (+ 23360us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2040779}
I20260504 13:56:18.676693 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 86) took 1791 ms. Trace:
I20260504 13:56:18.676977 10002 rpcz_store.cc:276] 0504 13:56:16.885464 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:16.885591 (+ 127us) service_pool.cc:224] Handling call
0504 13:56:18.676395 (+1790804us) ts_tablet_manager.cc:1151] Deleting tablet bd8cd1caaeb249e3bafee7f1f0d4df2f
0504 13:56:18.676437 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:18.676645 (+ 208us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1790238}
I20260504 13:56:18.679275 9473 leader_election.cc:304] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:18.680787 9473 leader_election.cc:304] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:18.681933 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 87) took 1753 ms. Trace:
I20260504 13:56:18.682346 10002 rpcz_store.cc:276] 0504 13:56:16.928936 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:16.929079 (+ 143us) service_pool.cc:224] Handling call
0504 13:56:18.681540 (+1752461us) ts_tablet_manager.cc:1151] Deleting tablet c34877a81d504f46a2f680c5d5ddb4f4
0504 13:56:18.681591 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:18.681870 (+ 279us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1751881}
I20260504 13:56:18.684995 9931 tablet_bootstrap.cc:492] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:18.685458 9931 ts_tablet_manager.cc:1403] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.101s user 0.009s sys 0.000s
I20260504 13:56:18.686684 10307 raft_consensus.cc:493] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:18.687155 10307 raft_consensus.cc:515] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.687969 9830 raft_consensus.cc:3060] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:18.687912 9931 raft_consensus.cc:359] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.688607 9931 raft_consensus.cc:385] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:18.688867 9931 raft_consensus.cc:740] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:18.690663 10307 leader_election.cc:290] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:18.690845 9931 consensus_queue.cc:260] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.692889 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46fdf0ffaf454689a705949bd39deb26" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:18.694056 9618 leader_election.cc:343] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:18.647092 9932 raft_consensus.cc:385] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:18.700812 9932 raft_consensus.cc:740] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:18.703701 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 88) took 1696 ms. Trace:
I20260504 13:56:18.705986 9931 ts_tablet_manager.cc:1434] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260504 13:56:18.706790 9931 tablet_bootstrap.cc:492] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:18.709537 10317 raft_consensus.cc:2804] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:18.710817 10318 raft_consensus.cc:2804] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:18.705390 9932 consensus_queue.cc:260] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.703997 10002 rpcz_store.cc:276] 0504 13:56:17.007601 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.007725 (+ 124us) service_pool.cc:224] Handling call
0504 13:56:18.703402 (+1695677us) ts_tablet_manager.cc:1151] Deleting tablet bd8cd1caaeb249e3bafee7f1f0d4df2f
0504 13:56:18.703442 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:18.703649 (+ 207us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1695145}
I20260504 13:56:18.717549 10002 tablet_replica.cc:333] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:18.724287 9932 ts_tablet_manager.cc:1434] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.080s user 0.004s sys 0.002s
I20260504 13:56:18.726588 10002 raft_consensus.cc:2243] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:18.727064 10002 raft_consensus.cc:2272] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:18.733531 9515 tablet_service.cc:1558] Processing DeleteTablet for tablet cdb86f6370bf4c5f9c488d85efb67ef7 with delete_type TABLET_DATA_DELETED (replaced by tablet 030b35c3409143108c34ab1f0ee1b96d) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:18.734982 9515 tablet_service.cc:1558] Processing DeleteTablet for tablet 5625163f6add4a20b1828905d90b5486 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:15 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:18.737906 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 91371bf848f5465ea2db0c090583382d (table default.Table_28052c777ee944b79b802d28b415c24e [id=994b65b95f66440283f852cd9dd9ae0c]) successfully deleted
W20260504 13:56:18.738649 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd8cd1caaeb249e3bafee7f1f0d4df2f already in progress: creating tablet
I20260504 13:56:18.738647 10317 raft_consensus.cc:697] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
W20260504 13:56:18.739698 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd8cd1caaeb249e3bafee7f1f0d4df2f already in progress: creating tablet
I20260504 13:56:18.739449 10317 consensus_queue.cc:237] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.744643 9932 tablet_bootstrap.cc:492] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:18.756199 9666 tablet_service.cc:1558] Processing DeleteTablet for tablet 39f080a0296b44988e8e51e68453a551 with delete_type TABLET_DATA_DELETED (replaced by tablet 274bb204041c4e1085395c7bd938aaf1) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:18.757138 9666 tablet_service.cc:1558] Processing DeleteTablet for tablet 65052a5c28c147a385144317e21eea41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:18.757917 9666 tablet_service.cc:1558] Processing DeleteTablet for tablet cdb86f6370bf4c5f9c488d85efb67ef7 with delete_type TABLET_DATA_DELETED (replaced by tablet 030b35c3409143108c34ab1f0ee1b96d) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:18.761005 10013 tablet_bootstrap.cc:654] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:18.766700 10307 raft_consensus.cc:493] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:18.767282 10307 raft_consensus.cc:515] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.775800 10273 raft_consensus.cc:493] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:18.776273 10273 raft_consensus.cc:515] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.782214 10307 leader_election.cc:290] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:18.783751 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e4239942f34fb081a026a35740e465" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:18.784509 10003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 97) took 1179 ms. Trace:
I20260504 13:56:18.784845 10003 rpcz_store.cc:276] 0504 13:56:17.604885 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.697420 (+ 92535us) service_pool.cc:224] Handling call
0504 13:56:18.450478 (+753058us) ts_tablet_manager.cc:1151] Deleting tablet 44017e2079be49258bab49325e3da1c1
0504 13:56:18.450518 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:18.608905 (+158387us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.668466 (+ 59561us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.784450 (+115984us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":746946}
W20260504 13:56:18.785427 9618 leader_election.cc:343] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:18.786329 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 44017e2079be49258bab49325e3da1c1 (table default.Table_f4a4640b6bb0432dad74376a7ed31181 [id=85b12280c48642b2a9cc8c51af037f2e]) successfully deleted
W20260504 13:56:18.786856 9618 leader_election.cc:343] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Not found: Tablet not found: d7e556f81ec940f1aa3c55b3a25f31ab
I20260504 13:56:18.787287 10003 tablet_replica.cc:333] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:18.789117 10273 leader_election.cc:290] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:18.789462 10319 raft_consensus.cc:493] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:18.789925 10319 raft_consensus.cc:515] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.792474 10319 leader_election.cc:290] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:18.793056 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdb86f6370bf4c5f9c488d85efb67ef7" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:18.794713 9474 leader_election.cc:343] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:18.795619 10003 raft_consensus.cc:2243] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:18.769291 10318 raft_consensus.cc:697] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:18.796800 10318 consensus_queue.cc:237] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.800042 9335 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_c5584202645f4ef78e4f6e00aa2bf2c1" } modify_external_catalogs: true
I20260504 13:56:18.800668 9335 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_c5584202645f4ef78e4f6e00aa2bf2c1" } modify_external_catalogs: true
W20260504 13:56:18.802129 9328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 60526.9us
I20260504 13:56:18.803294 9342 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_44cfda2157f6473cb8b53b4940f279cc" } new_table_name: "default.Table_03e728dbf91f4fd19afe882c3b4d7bda" modify_external_catalogs: true
I20260504 13:56:18.811601 9343 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 48b193e958b4414aad29dd2bcee1a61b (table default.Table_2c39b14515bb46f6a0b0f1252d8e273d [id=e9e56499fd2243919308927b3614ae8e]). Expected version 1 got 0
I20260504 13:56:18.812230 9343 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]). Expected version 1 got 0
I20260504 13:56:18.803426 9338 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_91c2d3ed5c5c42758c87f1f726af88c0" } modify_external_catalogs: true
I20260504 13:56:18.815155 9338 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_91c2d3ed5c5c42758c87f1f726af88c0" } modify_external_catalogs: true
I20260504 13:56:18.810631 10002 ts_tablet_manager.cc:1916] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:18.813019 10319 consensus_queue.cc:1048] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:18.817348 9932 tablet_bootstrap.cc:654] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:18.827091 9344 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 29) took 767 ms (client timeout 1000 ms). Trace:
I20260504 13:56:18.831444 10003 raft_consensus.cc:2272] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:18.834156 10003 ts_tablet_manager.cc:1916] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:18.836494 9944 ts_tablet_manager.cc:1929] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:18.837008 10013 tablet_bootstrap.cc:492] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:18.837415 10013 ts_tablet_manager.cc:1403] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.221s user 0.010s sys 0.000s
I20260504 13:56:18.848553 10013 raft_consensus.cc:359] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.849135 10013 raft_consensus.cc:385] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:18.849373 10013 raft_consensus.cc:740] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:18.869357 9339 master_service.cc:954] ReplaceTablet: received request to replace tablet 2b50573310a04f3694eeab2fef0fb540 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:18.880319 10002 ts_tablet_manager.cc:1929] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20260504 13:56:18.881412 10002 log.cc:1199] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/39f080a0296b44988e8e51e68453a551
I20260504 13:56:18.882090 10002 ts_tablet_manager.cc:1950] T 39f080a0296b44988e8e51e68453a551 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:18.891770 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 89) took 1845 ms. Trace:
I20260504 13:56:18.892055 10002 rpcz_store.cc:276] 0504 13:56:17.046030 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.046157 (+ 127us) service_pool.cc:224] Handling call
0504 13:56:18.717418 (+1671261us) ts_tablet_manager.cc:1151] Deleting tablet 39f080a0296b44988e8e51e68453a551
0504 13:56:18.717454 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:18.844640 (+127186us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.880195 (+ 35555us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.891724 (+ 11529us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8329,"tablet-delete.queue_time_us":1670681}
I20260504 13:56:18.893157 9931 tablet_bootstrap.cc:654] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:18.893465 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 39f080a0296b44988e8e51e68453a551 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]) successfully deleted
I20260504 13:56:18.894536 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 97) took 1176 ms. Trace:
I20260504 13:56:18.894912 10002 rpcz_store.cc:276] 0504 13:56:17.718303 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.718416 (+ 113us) service_pool.cc:224] Handling call
0504 13:56:18.894139 (+1175723us) ts_tablet_manager.cc:1151] Deleting tablet 3c43ff1d759a40d68344f25f066416ac
0504 13:56:18.894188 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:18.894474 (+ 286us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1175178}
W20260504 13:56:18.896070 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 3c43ff1d759a40d68344f25f066416ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c43ff1d759a40d68344f25f066416ac already in progress: creating tablet
I20260504 13:56:18.897799 10002 tablet_replica.cc:333] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:18.898553 10002 raft_consensus.cc:2243] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:18.899019 10002 raft_consensus.cc:2272] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:18.898777 10003 ts_tablet_manager.cc:1929] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:18.902344 10003 log.cc:1199] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/a43cdb767fd14ea09b4db4d2881b69aa
I20260504 13:56:18.903092 10003 ts_tablet_manager.cc:1950] T a43cdb767fd14ea09b4db4d2881b69aa P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
W20260504 13:56:18.827960 9344 rpcz_store.cc:269] 0504 13:56:18.059431 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.059578 (+ 147us) service_pool.cc:224] Handling call
0504 13:56:18.827025 (+767447us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:18.480762 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:18.481201 (+ 439us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:18.481234 (+ 33us) write_op.cc:435] Acquired schema lock
0504 13:56:18.481255 (+ 21us) tablet.cc:662] Decoding operations
0504 13:56:18.481579 (+ 324us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:18.481632 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:18.481665 (+ 33us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:18.481834 (+ 169us) tablet.cc:701] Row locks acquired
0504 13:56:18.481852 (+ 18us) write_op.cc:260] PREPARE: finished
0504 13:56:18.481971 (+ 119us) write_op.cc:270] Start()
0504 13:56:18.482065 (+ 94us) write_op.cc:276] Timestamp: P: 1777902978481917 usec, L: 0
0504 13:56:18.482087 (+ 22us) op_driver.cc:348] REPLICATION: starting
0504 13:56:18.482684 (+ 597us) log.cc:844] Serialized 533 byte log entry
0504 13:56:18.525116 (+ 42432us) op_driver.cc:464] REPLICATION: finished
0504 13:56:18.528419 (+ 3303us) write_op.cc:301] APPLY: starting
0504 13:56:18.528502 (+ 83us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:18.528630 (+ 128us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:18.528844 (+ 214us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:18.529094 (+ 250us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:18.529365 (+ 271us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:18.529433 (+ 68us) write_op.cc:312] APPLY: finished
0504 13:56:18.529690 (+ 257us) log.cc:844] Serialized 23 byte log entry
0504 13:56:18.530514 (+ 824us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:18.530612 (+ 98us) write_op.cc:454] Released schema lock
0504 13:56:18.530656 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:18.530968 (+ 312us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:18.531004 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":45119,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":3109,"apply.run_cpu_time_us":1742,"apply.run_wall_time_us":2500,"num_ops":1,"prepare.queue_time_us":8553,"prepare.run_cpu_time_us":2267,"prepare.run_wall_time_us":40346,"raft.queue_time_us":100,"raft.run_cpu_time_us":524,"raft.run_wall_time_us":524,"replication_time_us":42940}]]}
I20260504 13:56:18.905964 10003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 102) took 1022 ms. Trace:
I20260504 13:56:18.911134 9944 log.cc:1199] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/44017e2079be49258bab49325e3da1c1
I20260504 13:56:18.917686 10002 ts_tablet_manager.cc:1916] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:18.918246 10013 consensus_queue.cc:260] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.928694 10013 ts_tablet_manager.cc:1434] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.091s user 0.005s sys 0.002s
W20260504 13:56:18.932804 9342 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7cb6927537704cc4b17617cbdaa21ac1 has no consensus state
I20260504 13:56:18.933964 10318 raft_consensus.cc:493] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:18.934376 10318 raft_consensus.cc:515] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:18.936448 10318 leader_election.cc:290] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:18.944458 9342 master_service.cc:954] ReplaceTablet: received request to replace tablet 50e129980cd84f6ca22cbf2addb06230 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:18.847507 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7ab3b577b74a9c9a40cb15c94f8aa1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:18.947206 10013 tablet_bootstrap.cc:492] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:18.947439 9686 raft_consensus.cc:3060] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:18.875602 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdb86f6370bf4c5f9c488d85efb67ef7" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:18.951129 9683 raft_consensus.cc:2468] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
W20260504 13:56:18.877830 9620 leader_election.cc:343] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:18.842032 9932 tablet_bootstrap.cc:492] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:18.983629 9932 ts_tablet_manager.cc:1403] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.253s user 0.009s sys 0.000s
I20260504 13:56:18.984185 10013 tablet_bootstrap.cc:654] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:18.984642 10002 ts_tablet_manager.cc:1929] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:18.984968 10002 log.cc:1199] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/44017e2079be49258bab49325e3da1c1
I20260504 13:56:18.985642 10002 ts_tablet_manager.cc:1950] T 44017e2079be49258bab49325e3da1c1 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:18.986560 9932 raft_consensus.cc:359] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:18.987655 9620 leader_election.cc:304] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:18.988329 10273 raft_consensus.cc:2749] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:18.989131 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 111328us
I20260504 13:56:18.989526 9932 raft_consensus.cc:385] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:18.989864 9932 raft_consensus.cc:740] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:18.990540 9932 consensus_queue.cc:260] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:18.997833 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 95) took 1284 ms. Trace:
I20260504 13:56:18.998126 10002 rpcz_store.cc:276] 0504 13:56:17.713614 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.719083 (+ 5469us) service_pool.cc:224] Handling call
0504 13:56:18.897629 (+1178546us) ts_tablet_manager.cc:1151] Deleting tablet 44017e2079be49258bab49325e3da1c1
0504 13:56:18.897678 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:18.955641 (+ 57963us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.984555 (+ 28914us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.997780 (+ 13225us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1177916}
I20260504 13:56:18.999336 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 102) took 1063 ms. Trace:
I20260504 13:56:18.999380 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 44017e2079be49258bab49325e3da1c1 (table default.Table_f4a4640b6bb0432dad74376a7ed31181 [id=85b12280c48642b2a9cc8c51af037f2e]) successfully deleted
I20260504 13:56:18.999639 10002 rpcz_store.cc:276] 0504 13:56:17.936383 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.936768 (+ 385us) service_pool.cc:224] Handling call
0504 13:56:18.999056 (+1062288us) ts_tablet_manager.cc:1151] Deleting tablet 14b7f1e834d0439c89abb180bd13853d
0504 13:56:18.999099 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:18.999276 (+ 177us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061744}
W20260504 13:56:19.001303 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 14b7f1e834d0439c89abb180bd13853d because the tablet was not found. No further retry: Not found: Tablet not found: 14b7f1e834d0439c89abb180bd13853d
I20260504 13:56:19.005239 9340 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_7aba2a5065e04c10a7116b73aabdf238" } new_table_name: "default.Table_50724ddf9a35434385ad5cecc66746d9" modify_external_catalogs: true
I20260504 13:56:19.009109 9932 ts_tablet_manager.cc:1434] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260504 13:56:19.011974 10002 tablet_replica.cc:333] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:19.011888 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_8cae213b77994b9fba63b8a7c5f27249"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:19.013319 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cae213b77994b9fba63b8a7c5f27249 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:19.015796 9336 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_737cc12f07144a6c92dc2974693a649d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:19.018147 9336 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_737cc12f07144a6c92dc2974693a649d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:19.012645 10002 raft_consensus.cc:2243] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:19.019615 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_5bc32a610edd49ab9641c3726f53f5c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:19.020354 10002 raft_consensus.cc:2272] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:19.021627 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bc32a610edd49ab9641c3726f53f5c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:19.022053 10002 ts_tablet_manager.cc:1916] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:19.032446 10013 tablet_bootstrap.cc:492] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:19.032869 10013 ts_tablet_manager.cc:1403] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.103s user 0.009s sys 0.004s
I20260504 13:56:19.035665 10013 raft_consensus.cc:359] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.038345 9686 raft_consensus.cc:2468] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:19.038805 10013 raft_consensus.cc:385] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.042374 10013 raft_consensus.cc:740] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.043010 10013 consensus_queue.cc:260] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.044895 10013 ts_tablet_manager.cc:1434] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260504 13:56:18.938644 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65052a5c28c147a385144317e21eea41" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:18.952226 9931 tablet_bootstrap.cc:492] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:19.047785 9931 ts_tablet_manager.cc:1403] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.341s user 0.007s sys 0.005s
I20260504 13:56:18.959950 10246 raft_consensus.cc:493] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5beceb6317444247b31b44a6581707c6)
I20260504 13:56:19.048187 10013 tablet_bootstrap.cc:492] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:19.048149 10246 raft_consensus.cc:515] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.049973 9931 raft_consensus.cc:359] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.050598 9931 raft_consensus.cc:385] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.050822 9931 raft_consensus.cc:740] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
W20260504 13:56:19.051937 9474 leader_election.cc:343] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.052601 9931 consensus_queue.cc:260] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.055675 9931 ts_tablet_manager.cc:1434] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260504 13:56:19.056445 9931 tablet_bootstrap.cc:492] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:19.060128 10246 leader_election.cc:290] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:19.062538 9931 tablet_bootstrap.cc:654] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.062856 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de639fda874248f0853e74ee60aaf758" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:19.070089 10003 rpcz_store.cc:276] 0504 13:56:17.883547 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.883685 (+ 138us) service_pool.cc:224] Handling call
0504 13:56:18.787087 (+903402us) ts_tablet_manager.cc:1151] Deleting tablet a43cdb767fd14ea09b4db4d2881b69aa
0504 13:56:18.787157 (+ 70us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:18.866736 (+ 79579us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.898666 (+ 31930us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.905917 (+ 7251us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4062,"tablet-delete.queue_time_us":902882}
I20260504 13:56:19.073560 10013 tablet_bootstrap.cc:654] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.073655 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet a43cdb767fd14ea09b4db4d2881b69aa (table default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]) successfully deleted
I20260504 13:56:19.074882 10002 ts_tablet_manager.cc:1929] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:19.075229 10002 log.cc:1199] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/0867df9ea115418193b6d9d3fc251923
I20260504 13:56:19.075510 10003 tablet_replica.cc:333] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:19.039885 10273 consensus_queue.cc:1048] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:19.040613 9620 consensus_peers.cc:597] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3793fe8f1ec43ad809403c4593f5ef6. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:19.010699 9932 tablet_bootstrap.cc:492] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:19.061280 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de639fda874248f0853e74ee60aaf758" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
W20260504 13:56:19.077303 9620 leader_election.cc:343] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.083683 9620 leader_election.cc:304] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.075907 10002 ts_tablet_manager.cc:1950] T 0867df9ea115418193b6d9d3fc251923 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:19.084321 10273 raft_consensus.cc:2749] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:19.085479 10319 raft_consensus.cc:493] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 13:56:19.086095 9620 leader_election.cc:343] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.085880 10319 raft_consensus.cc:515] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.086701 9620 leader_election.cc:304] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.087702 10319 leader_election.cc:290] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.087684 10273 raft_consensus.cc:2749] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:19.069759 9944 ts_tablet_manager.cc:1950] T 44017e2079be49258bab49325e3da1c1 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:19.085870 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 101) took 1149 ms. Trace:
I20260504 13:56:19.094751 10002 rpcz_store.cc:276] 0504 13:56:17.936063 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.937288 (+ 1225us) service_pool.cc:224] Handling call
0504 13:56:19.011837 (+1074549us) ts_tablet_manager.cc:1151] Deleting tablet 0867df9ea115418193b6d9d3fc251923
0504 13:56:19.011875 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:19.050535 (+ 38660us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:19.074773 (+ 24238us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:19.085815 (+ 11042us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1074087}
I20260504 13:56:19.100243 10002 tablet_replica.cc:333] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:19.100912 10002 raft_consensus.cc:2243] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:19.101383 10002 raft_consensus.cc:2272] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:19.102979 10002 ts_tablet_manager.cc:1916] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:19.078809 9768 leader_election.cc:304] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:19.105078 9931 tablet_bootstrap.cc:492] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:19.089927 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46fdf0ffaf454689a705949bd39deb26" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:19.105438 9931 ts_tablet_manager.cc:1403] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.000s
I20260504 13:56:19.107870 9931 raft_consensus.cc:359] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.108457 9931 raft_consensus.cc:385] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.108680 9931 raft_consensus.cc:740] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.109243 9931 consensus_queue.cc:260] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.109856 10326 raft_consensus.cc:2749] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:19.105834 10013 tablet_bootstrap.cc:492] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:19.115513 10013 ts_tablet_manager.cc:1403] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.004s
I20260504 13:56:19.117825 10013 raft_consensus.cc:359] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.118407 10013 raft_consensus.cc:385] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.118649 10013 raft_consensus.cc:740] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
W20260504 13:56:19.108723 9474 leader_election.cc:343] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.119498 10003 raft_consensus.cc:2243] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:19.119460 10013 consensus_queue.cc:260] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.120012 10003 raft_consensus.cc:2272] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:19.123476 10013 ts_tablet_manager.cc:1434] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20260504 13:56:19.124270 10013 tablet_bootstrap.cc:492] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:19.138980 10319 raft_consensus.cc:493] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:19.139412 10319 raft_consensus.cc:515] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.141079 10319 leader_election.cc:290] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.141258 9931 ts_tablet_manager.cc:1434] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20260504 13:56:19.142706 9931 tablet_bootstrap.cc:492] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:19.143500 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b50573310a04f3694eeab2fef0fb540" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:19.144614 9474 leader_election.cc:343] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.150458 10002 ts_tablet_manager.cc:1929] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:19.150774 10002 log.cc:1199] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/b3793fe8f1ec43ad809403c4593f5ef6
I20260504 13:56:19.151866 9338 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet afee197774db411d97468913a207703a
I20260504 13:56:19.152197 9931 tablet_bootstrap.cc:654] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.153611 9515 tablet_service.cc:1558] Processing DeleteTablet for tablet afee197774db411d97468913a207703a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:18 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:19.156073 10013 tablet_bootstrap.cc:654] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.157655 9810 tablet_service.cc:1558] Processing DeleteTablet for tablet afee197774db411d97468913a207703a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:18 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:19.169790 10002 ts_tablet_manager.cc:1950] T b3793fe8f1ec43ad809403c4593f5ef6 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:19.171619 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 103) took 1235 ms. Trace:
I20260504 13:56:19.171912 10002 rpcz_store.cc:276] 0504 13:56:17.936646 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.937406 (+ 760us) service_pool.cc:224] Handling call
0504 13:56:19.100096 (+1162690us) ts_tablet_manager.cc:1151] Deleting tablet b3793fe8f1ec43ad809403c4593f5ef6
0504 13:56:19.100139 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:19.130693 (+ 30554us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:19.150350 (+ 19657us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:19.171554 (+ 21204us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1162170}
I20260504 13:56:19.172776 10013 tablet_bootstrap.cc:492] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:19.172957 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet b3793fe8f1ec43ad809403c4593f5ef6 (table default.Table_61add6c7b0dd46b08c0e39ad6d697e8c [id=1749c1a022544b1185fb3d4a4a530afd]) successfully deleted
I20260504 13:56:19.173153 10013 ts_tablet_manager.cc:1403] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.005s
I20260504 13:56:19.175511 10013 raft_consensus.cc:359] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.176213 10013 raft_consensus.cc:385] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.176460 10013 raft_consensus.cc:740] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
W20260504 13:56:19.176649 9330 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 70caac2e256d419dbc2e70cb5dfffe97 (table default.Table_c5584202645f4ef78e4f6e00aa2bf2c1 [id=2665806fb81f4eda8981fede847cbfcf], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 70caac2e256d419dbc2e70cb5dfffe97 consensus state has no leader
I20260504 13:56:19.181031 9931 tablet_bootstrap.cc:492] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:19.181155 9338 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet de639fda874248f0853e74ee60aaf758 (table default.Table_0d4595642a604b8f8f57d3eadbbacacb [id=666099fdb1a94091ac6fd99cf58f18e2]). Expected version 1 got 0
I20260504 13:56:19.181514 9931 ts_tablet_manager.cc:1403] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.000s
I20260504 13:56:19.184840 9931 raft_consensus.cc:359] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.185416 9931 raft_consensus.cc:385] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.185621 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 88) took 1318 ms. Trace:
I20260504 13:56:19.185699 9931 raft_consensus.cc:740] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.185998 9944 rpcz_store.cc:276] 0504 13:56:17.867010 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.867128 (+ 118us) service_pool.cc:224] Handling call
0504 13:56:18.373340 (+506212us) ts_tablet_manager.cc:1151] Deleting tablet 44017e2079be49258bab49325e3da1c1
0504 13:56:18.373387 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:18.612427 (+239040us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:18.836388 (+223961us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:19.185560 (+349172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24402,"tablet-delete.queue_time_us":505628}
I20260504 13:56:19.186527 9931 consensus_queue.cc:260] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.187289 10002 tablet_replica.cc:333] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:19.187436 10013 consensus_queue.cc:260] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.188035 10002 raft_consensus.cc:2243] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:19.187964 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 89) took 1318 ms. Trace:
I20260504 13:56:19.188411 9931 ts_tablet_manager.cc:1434] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
W20260504 13:56:19.196516 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 105381us
I20260504 13:56:19.198671 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46fdf0ffaf454689a705949bd39deb26" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:19.200693 10013 ts_tablet_manager.cc:1434] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.027s user 0.004s sys 0.000s
I20260504 13:56:19.200690 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e4239942f34fb081a026a35740e465" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:19.201577 10013 tablet_bootstrap.cc:492] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:19.205540 10002 pending_rounds.cc:70] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00: Trying to abort 1 pending ops.
I20260504 13:56:19.206365 10002 pending_rounds.cc:77] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290600933900288 op_type: NO_OP noop_request { }
I20260504 13:56:19.206698 10002 raft_consensus.cc:2889] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:19.207079 10002 raft_consensus.cc:2272] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:19.211678 9618 consensus_peers.cc:597] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0867df9ea115418193b6d9d3fc251923. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:19.213934 10002 ts_tablet_manager.cc:1916] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:19.221645 9932 tablet_bootstrap.cc:654] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.222801 10013 tablet_bootstrap.cc:654] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.248888 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_e116f9abbb23419d9a3365f7d0e17db9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:19.251427 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e116f9abbb23419d9a3365f7d0e17db9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:19.256742 10013 tablet_bootstrap.cc:492] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:19.257112 10013 ts_tablet_manager.cc:1403] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.000s
I20260504 13:56:19.259332 10013 raft_consensus.cc:359] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.259902 10013 raft_consensus.cc:385] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.260156 10013 raft_consensus.cc:740] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.266572 10013 consensus_queue.cc:260] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.271724 10336 raft_consensus.cc:493] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:19.272212 10013 ts_tablet_manager.cc:1434] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260504 13:56:19.272511 10002 ts_tablet_manager.cc:1929] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:19.272192 10336 raft_consensus.cc:515] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.273962 10336 leader_election.cc:290] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:19.272841 10002 log.cc:1199] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/a43cdb767fd14ea09b4db4d2881b69aa
I20260504 13:56:19.281800 10002 ts_tablet_manager.cc:1950] T a43cdb767fd14ea09b4db4d2881b69aa P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:19.283576 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 104) took 1229 ms. Trace:
I20260504 13:56:19.283852 10002 rpcz_store.cc:276] 0504 13:56:18.054327 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.054446 (+ 119us) service_pool.cc:224] Handling call
0504 13:56:19.187123 (+1132677us) ts_tablet_manager.cc:1151] Deleting tablet a43cdb767fd14ea09b4db4d2881b69aa
0504 13:56:19.187183 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:19.245510 (+ 58327us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:19.272399 (+ 26889us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:19.283523 (+ 11124us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1132127}
I20260504 13:56:19.273036 10013 tablet_bootstrap.cc:492] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:19.284997 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet a43cdb767fd14ea09b4db4d2881b69aa (table default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]) successfully deleted
I20260504 13:56:19.286794 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "baf07a970a7c4a35b3c849b06d6c88d0" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:19.300030 10002 tablet_replica.cc:333] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:19.300980 10002 raft_consensus.cc:2243] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:19.301532 10002 raft_consensus.cc:2272] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:19.302588 9473 leader_election.cc:304] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.304701 10319 raft_consensus.cc:2804] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:19.305017 10319 raft_consensus.cc:493] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:19.305321 10319 raft_consensus.cc:3060] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.305711 9931 tablet_bootstrap.cc:492] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:19.310607 9931 tablet_bootstrap.cc:654] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.311134 10318 raft_consensus.cc:493] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:19.311542 10318 raft_consensus.cc:515] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.313341 10318 leader_election.cc:290] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.315369 10002 ts_tablet_manager.cc:1916] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:19.315682 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c43ff1d759a40d68344f25f066416ac" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:19.316504 10003 ts_tablet_manager.cc:1916] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260504 13:56:19.317018 9474 leader_election.cc:343] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.188488 9944 rpcz_store.cc:276] 0504 13:56:17.869426 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.869557 (+ 131us) service_pool.cc:224] Handling call
0504 13:56:19.187376 (+1317819us) ts_tablet_manager.cc:1151] Deleting tablet 03818e34ac0545be880c873cf82cd4ea
0504 13:56:19.187417 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:19.187914 (+ 497us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1317292}
I20260504 13:56:19.324971 10319 raft_consensus.cc:515] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.328456 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdb86f6370bf4c5f9c488d85efb67ef7" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:19.329066 9829 raft_consensus.cc:3060] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.335798 9473 leader_election.cc:304] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.336658 10318 raft_consensus.cc:2804] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:19.337319 10318 raft_consensus.cc:697] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:19.338629 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 44017e2079be49258bab49325e3da1c1 (table default.Table_f4a4640b6bb0432dad74376a7ed31181 [id=85b12280c48642b2a9cc8c51af037f2e]) successfully deleted
I20260504 13:56:19.338512 10318 consensus_queue.cc:237] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.340613 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 90) took 1470 ms. Trace:
I20260504 13:56:19.340943 9944 rpcz_store.cc:276] 0504 13:56:17.869773 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.870176 (+ 403us) service_pool.cc:224] Handling call
0504 13:56:19.322023 (+1451847us) ts_tablet_manager.cc:1151] Deleting tablet b4dca0cadd5e400eab1365c03a8de4f4
0504 13:56:19.322064 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:19.340559 (+ 18495us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1451349}
I20260504 13:56:19.341931 9944 tablet_replica.cc:333] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:19.342682 9829 raft_consensus.cc:2468] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:19.343218 9944 raft_consensus.cc:2243] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:19.343709 9474 leader_election.cc:304] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:19.344449 10317 raft_consensus.cc:2804] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:19.354040 9336 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_597ab174935c4601914d71c16e0c2620" } new_table_name: "default.Table_f5ef8c8f56c74734b79e268bd64f85e3" modify_external_catalogs: true
I20260504 13:56:19.323107 10013 tablet_bootstrap.cc:654] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.369251 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "baf07a970a7c4a35b3c849b06d6c88d0" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:19.370121 9686 raft_consensus.cc:2393] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:19.371265 9768 leader_election.cc:304] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:19.372062 10336 raft_consensus.cc:3060] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.374047 10273 raft_consensus.cc:493] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:19.374495 10273 raft_consensus.cc:515] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.376441 10273 leader_election.cc:290] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.377668 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34877a81d504f46a2f680c5d5ddb4f4" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:19.379040 9618 leader_election.cc:343] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.380168 9665 tablet_service.cc:1558] Processing DeleteTablet for tablet 07a428049fdb4a68aae729400e2d9674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:14 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:19.378576 10319 leader_election.cc:290] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.381896 10336 raft_consensus.cc:2749] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:19.384653 10273 raft_consensus.cc:493] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:19.379006 10002 ts_tablet_manager.cc:1929] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:19.385514 10273 raft_consensus.cc:515] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.386323 10002 log.cc:1199] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/eb7f58645c76441589368072274ebc8f
I20260504 13:56:19.387015 10002 ts_tablet_manager.cc:1950] T eb7f58645c76441589368072274ebc8f P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
W20260504 13:56:19.388880 9618 leader_election.cc:343] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Not found: Tablet not found: 14b7f1e834d0439c89abb180bd13853d
I20260504 13:56:19.389338 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet eb7f58645c76441589368072274ebc8f (table default.Table_d5cc86d81fb043319576af0a39c643a8 [id=7af4adcf5ee44bad89afa2e4ded416fc]) successfully deleted
I20260504 13:56:19.389623 10273 leader_election.cc:290] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.385929 9944 raft_consensus.cc:2272] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:19.391645 9944 ts_tablet_manager.cc:1916] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260504 13:56:19.400319 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 0b7a71dbcc694e1a97019188cfb2c672 because the tablet was not found. No further retry: Not found: Tablet not found: 0b7a71dbcc694e1a97019188cfb2c672
W20260504 13:56:19.402503 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet d7e556f81ec940f1aa3c55b3a25f31ab because the tablet was not found. No further retry: Not found: Tablet not found: d7e556f81ec940f1aa3c55b3a25f31ab
I20260504 13:56:19.403391 10343 tablet.cc:1724] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:19.403947 10002 tablet_replica.cc:333] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:19.405020 10317 raft_consensus.cc:697] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:19.405887 10317 consensus_queue.cc:237] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.408378 10002 raft_consensus.cc:2243] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:19.409071 10002 raft_consensus.cc:2272] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:19.413771 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 03818e34ac0545be880c873cf82cd4ea because the tablet was not found. No further retry: Not found: Tablet not found: 03818e34ac0545be880c873cf82cd4ea
I20260504 13:56:19.414080 10013 tablet_bootstrap.cc:492] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:19.414422 10013 ts_tablet_manager.cc:1403] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.142s user 0.009s sys 0.000s
W20260504 13:56:19.414384 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet b4dca0cadd5e400eab1365c03a8de4f4 because the tablet was not found. No further retry: Not found: Tablet not found: b4dca0cadd5e400eab1365c03a8de4f4
I20260504 13:56:19.414882 10002 ts_tablet_manager.cc:1916] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:19.416599 10013 raft_consensus.cc:359] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.417152 10013 raft_consensus.cc:385] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.417393 10013 raft_consensus.cc:740] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.392905 9665 tablet_service.cc:1558] Processing DeleteTablet for tablet afee197774db411d97468913a207703a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:18 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:19.394117 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65052a5c28c147a385144317e21eea41" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:19.420460 9686 raft_consensus.cc:2468] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:19.419821 10013 consensus_queue.cc:260] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.424485 10013 ts_tablet_manager.cc:1434] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.010s user 0.000s sys 0.005s
I20260504 13:56:19.432715 9341 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_05a212a8397548bd94b840d6b4cf4fd7" } modify_external_catalogs: true
I20260504 13:56:19.433241 9341 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_05a212a8397548bd94b840d6b4cf4fd7" } modify_external_catalogs: true
I20260504 13:56:19.416160 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34877a81d504f46a2f680c5d5ddb4f4" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:19.435382 10013 tablet_bootstrap.cc:492] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:19.436461 9931 tablet_bootstrap.cc:492] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:19.436844 9931 ts_tablet_manager.cc:1403] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.131s user 0.021s sys 0.012s
I20260504 13:56:19.411500 9932 tablet_bootstrap.cc:492] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:19.412555 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46fdf0ffaf454689a705949bd39deb26" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:19.443149 9684 raft_consensus.cc:2468] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:19.413081 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdb86f6370bf4c5f9c488d85efb67ef7" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:19.444294 9682 raft_consensus.cc:3060] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.413216 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c43ff1d759a40d68344f25f066416ac" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:19.443156 9932 ts_tablet_manager.cc:1403] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.433s user 0.006s sys 0.004s
I20260504 13:56:19.417596 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b50573310a04f3694eeab2fef0fb540" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:19.447927 9683 raft_consensus.cc:2468] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:19.447425 9685 raft_consensus.cc:2468] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
W20260504 13:56:19.418156 9618 consensus_peers.cc:597] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb7f58645c76441589368072274ebc8f. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:19.449141 9931 raft_consensus.cc:359] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.449708 9931 raft_consensus.cc:385] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.449960 9931 raft_consensus.cc:740] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.450834 9931 consensus_queue.cc:260] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.452579 9931 ts_tablet_manager.cc:1434] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260504 13:56:19.454139 9682 raft_consensus.cc:2468] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:19.456197 9932 raft_consensus.cc:359] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:19.456789 9932 raft_consensus.cc:385] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.457036 9932 raft_consensus.cc:740] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.457288 10013 tablet_bootstrap.cc:654] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.464354 9931 tablet_bootstrap.cc:492] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:19.466488 9620 leader_election.cc:343] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.466977 9620 leader_election.cc:304] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.467824 10273 raft_consensus.cc:2749] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:19.472985 10013 tablet_bootstrap.cc:492] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
W20260504 13:56:19.473091 9620 leader_election.cc:343] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Not found: Tablet not found: d7e556f81ec940f1aa3c55b3a25f31ab
I20260504 13:56:19.473348 10013 ts_tablet_manager.cc:1403] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.005s
I20260504 13:56:19.473435 9620 leader_election.cc:304] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.474251 10273 raft_consensus.cc:2749] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:19.475499 10013 raft_consensus.cc:359] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.476044 10013 raft_consensus.cc:385] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.476269 10013 raft_consensus.cc:740] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.483451 9932 consensus_queue.cc:260] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:19.487020 9340 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de6714ee86d64112baf13dd1b2b87582 has no consensus state
I20260504 13:56:19.487293 9931 tablet_bootstrap.cc:654] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.487018 10013 consensus_queue.cc:260] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.490406 10002 ts_tablet_manager.cc:1929] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:19.490717 10002 log.cc:1199] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/69ed77782cd34274932957106ce26766
I20260504 13:56:19.491015 10013 ts_tablet_manager.cc:1434] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.017s user 0.003s sys 0.003s
I20260504 13:56:19.494216 10002 ts_tablet_manager.cc:1950] T 69ed77782cd34274932957106ce26766 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:19.495280 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_203b5e06868048d2a803bf466f3959d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:19.496376 9932 ts_tablet_manager.cc:1434] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.049s user 0.004s sys 0.000s
W20260504 13:56:19.496919 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_203b5e06868048d2a803bf466f3959d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:19.498790 9932 tablet_bootstrap.cc:492] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:19.498809 10013 tablet_bootstrap.cc:492] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:19.485582 9620 leader_election.cc:343] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.503499 9620 leader_election.cc:304] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.504082 10273 raft_consensus.cc:2749] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:19.506016 9620 consensus_peers.cc:597] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:19.506913 9620 leader_election.cc:343] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Not found: Tablet not found: 14b7f1e834d0439c89abb180bd13853d
I20260504 13:56:19.507306 9620 leader_election.cc:304] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.511327 10273 raft_consensus.cc:2749] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:19.504436 10002 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 115) took 1051 ms. Trace:
I20260504 13:56:19.515563 10002 rpcz_store.cc:276] 0504 13:56:18.453094 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.453192 (+ 98us) service_pool.cc:224] Handling call
0504 13:56:19.403819 (+950627us) ts_tablet_manager.cc:1151] Deleting tablet 69ed77782cd34274932957106ce26766
0504 13:56:19.403854 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:19.445695 (+ 41841us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:19.490295 (+ 44600us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:19.504370 (+ 14075us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":940726}
I20260504 13:56:19.516705 10013 tablet_bootstrap.cc:654] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.516675 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 69ed77782cd34274932957106ce26766 (table default.Table_cdaaea60f9354320bc4cfee6729ba12f [id=55db3ce6e7694cca94a1705fb1c0e90e]) successfully deleted
I20260504 13:56:19.520448 9931 tablet_bootstrap.cc:492] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:19.520920 9931 ts_tablet_manager.cc:1403] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.004s
I20260504 13:56:19.526360 10013 tablet_bootstrap.cc:492] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:19.526708 10013 ts_tablet_manager.cc:1403] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.000s
I20260504 13:56:19.528910 10013 raft_consensus.cc:359] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.529464 10013 raft_consensus.cc:385] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.529716 10013 raft_consensus.cc:740] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.532974 9931 raft_consensus.cc:359] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.533555 9931 raft_consensus.cc:385] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.533779 9931 raft_consensus.cc:740] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.534500 10317 raft_consensus.cc:493] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:19.534897 10317 raft_consensus.cc:515] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.538828 9931 consensus_queue.cc:260] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.539503 10317 leader_election.cc:290] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.540141 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afee197774db411d97468913a207703a" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:19.540699 9834 raft_consensus.cc:2468] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
W20260504 13:56:19.541342 9620 leader_election.cc:343] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.541632 9474 leader_election.cc:304] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:19.541705 9620 leader_election.cc:304] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.542383 10317 raft_consensus.cc:2804] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:19.542701 10317 raft_consensus.cc:493] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:19.542901 9931 ts_tablet_manager.cc:1434] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20260504 13:56:19.542984 10317 raft_consensus.cc:3060] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.543746 9931 tablet_bootstrap.cc:492] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:19.544895 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_1d9e75eee18c499cafb96717ac8bd724"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:19.546746 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d9e75eee18c499cafb96717ac8bd724 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:19.550120 10317 raft_consensus.cc:515] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.550521 10273 raft_consensus.cc:2749] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:19.552397 9473 leader_election.cc:304] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.553373 10345 raft_consensus.cc:2804] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:19.553664 10345 raft_consensus.cc:493] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:19.553974 10345 raft_consensus.cc:3060] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.554327 10002 tablet_replica.cc:333] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:19.555153 10002 raft_consensus.cc:2243] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:19.556223 10002 pending_rounds.cc:70] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00: Trying to abort 1 pending ops.
I20260504 13:56:19.556151 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afee197774db411d97468913a207703a" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:19.556430 10002 pending_rounds.cc:77] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290603656052736 op_type: NO_OP noop_request { }
I20260504 13:56:19.556720 10002 raft_consensus.cc:2889] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:19.556860 9834 raft_consensus.cc:3060] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.557029 10319 raft_consensus.cc:493] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:19.557446 10319 raft_consensus.cc:515] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.561348 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a428049fdb4a68aae729400e2d9674" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:19.559115 10319 leader_election.cc:290] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
W20260504 13:56:19.562590 9474 leader_election.cc:343] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.566618 10317 leader_election.cc:290] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.566205 9834 raft_consensus.cc:2468] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:19.557049 10002 raft_consensus.cc:2272] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:19.567988 9474 leader_election.cc:304] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:19.570261 10002 ts_tablet_manager.cc:1916] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:19.572585 9931 tablet_bootstrap.cc:654] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.575342 10345 raft_consensus.cc:515] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.577601 9473 leader_election.cc:304] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.578574 10319 raft_consensus.cc:2804] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:19.578977 10319 raft_consensus.cc:697] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
W20260504 13:56:19.579099 9336 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7eaa40eb4b1c425a9edfb7eaba953b65 has no consensus state
I20260504 13:56:19.580508 10318 raft_consensus.cc:2804] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:19.581337 10318 raft_consensus.cc:493] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:19.581631 10318 raft_consensus.cc:3060] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.582027 10013 consensus_queue.cc:260] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.579706 10319 consensus_queue.cc:237] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.584004 10013 ts_tablet_manager.cc:1434] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.057s user 0.006s sys 0.000s
I20260504 13:56:19.589847 10002 ts_tablet_manager.cc:1929] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:19.590176 10002 log.cc:1199] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/cdb86f6370bf4c5f9c488d85efb67ef7
I20260504 13:56:19.590958 10002 ts_tablet_manager.cc:1950] T cdb86f6370bf4c5f9c488d85efb67ef7 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:19.590293 10318 raft_consensus.cc:515] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.595410 9931 tablet_bootstrap.cc:492] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:19.595911 9931 ts_tablet_manager.cc:1403] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.003s
I20260504 13:56:19.584779 10013 tablet_bootstrap.cc:492] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:19.598659 9931 raft_consensus.cc:359] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.600605 10345 leader_election.cc:290] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.602371 9473 leader_election.cc:304] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.603008 9931 raft_consensus.cc:385] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.603313 9931 raft_consensus.cc:740] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.601547 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65052a5c28c147a385144317e21eea41" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:19.603694 9473 leader_election.cc:304] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.603993 9834 raft_consensus.cc:3060] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.604841 10317 raft_consensus.cc:2804] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:19.604924 9931 consensus_queue.cc:260] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.605175 10317 raft_consensus.cc:493] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:19.606113 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46fdf0ffaf454689a705949bd39deb26" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:19.606415 10319 raft_consensus.cc:2804] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:19.606581 9828 raft_consensus.cc:3060] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.606717 10319 raft_consensus.cc:493] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:19.607232 10319 raft_consensus.cc:3060] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.610334 9341 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet e11ff0ee553045918ac2e8de80a07da7
I20260504 13:56:19.616246 9828 raft_consensus.cc:2468] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:19.605607 10317 raft_consensus.cc:3060] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.617319 9474 leader_election.cc:304] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:19.618420 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_780e641042db4fd5aeb8af8b1aa08c72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:19.618803 9931 ts_tablet_manager.cc:1434] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.023s user 0.003s sys 0.003s
I20260504 13:56:19.619709 9931 tablet_bootstrap.cc:492] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:19.619894 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_780e641042db4fd5aeb8af8b1aa08c72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:19.622025 10273 raft_consensus.cc:493] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:19.622433 10273 raft_consensus.cc:515] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:19.624550 10273 leader_election.cc:290] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:19.625633 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be795f04dd54cf19869c937b91d0ddf" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:19.626829 9618 leader_election.cc:343] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.628634 9932 tablet_bootstrap.cc:654] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.630200 10317 raft_consensus.cc:515] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.631067 10003 ts_tablet_manager.cc:1929] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:19.632586 10003 log.cc:1199] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/eb7f58645c76441589368072274ebc8f
I20260504 13:56:19.633124 9944 ts_tablet_manager.cc:1929] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:19.633471 9944 log.cc:1199] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/0867df9ea115418193b6d9d3fc251923
I20260504 13:56:19.634126 9944 ts_tablet_manager.cc:1950] T 0867df9ea115418193b6d9d3fc251923 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:19.634164 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet cdb86f6370bf4c5f9c488d85efb67ef7 (table default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]) successfully deleted
I20260504 13:56:19.639082 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afee197774db411d97468913a207703a" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:19.639606 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65052a5c28c147a385144317e21eea41" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
W20260504 13:56:19.639803 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 5625163f6add4a20b1828905d90b5486 because the tablet was not found. No further retry: Not found: Tablet not found: 5625163f6add4a20b1828905d90b5486
I20260504 13:56:19.640097 9686 raft_consensus.cc:3060] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.640771 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afee197774db411d97468913a207703a" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:19.640555 10002 tablet_replica.cc:333] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:19.641786 10002 raft_consensus.cc:2243] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:19.641950 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a428049fdb4a68aae729400e2d9674" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:19.642408 10002 pending_rounds.cc:70] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00: Trying to abort 1 pending ops.
I20260504 13:56:19.642477 9682 raft_consensus.cc:2468] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:19.642602 10002 pending_rounds.cc:77] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290604379815936 op_type: NO_OP noop_request { }
I20260504 13:56:19.642879 10002 raft_consensus.cc:2889] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:19.639628 9685 raft_consensus.cc:2468] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:19.644021 10013 tablet_bootstrap.cc:654] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.644132 9834 raft_consensus.cc:2468] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:19.646193 9931 tablet_bootstrap.cc:654] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.646287 9474 leader_election.cc:304] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:19.647385 9686 raft_consensus.cc:2468] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:19.648453 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46fdf0ffaf454689a705949bd39deb26" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:19.649032 9686 raft_consensus.cc:3060] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.653095 10002 raft_consensus.cc:2272] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:19.633353 10003 ts_tablet_manager.cc:1950] T eb7f58645c76441589368072274ebc8f P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:19.654094 10317 leader_election.cc:290] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.654603 10317 raft_consensus.cc:2804] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:19.654968 10317 raft_consensus.cc:697] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:19.655876 10003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 110) took 1265 ms. Trace:
I20260504 13:56:19.656231 10003 rpcz_store.cc:276] 0504 13:56:18.390610 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.390840 (+ 230us) service_pool.cc:224] Handling call
0504 13:56:19.074971 (+684131us) ts_tablet_manager.cc:1151] Deleting tablet eb7f58645c76441589368072274ebc8f
0504 13:56:19.075023 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:19.506512 (+431489us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:19.630946 (+124434us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:19.655788 (+ 24842us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":656304}
I20260504 13:56:19.657382 10318 leader_election.cc:290] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.658006 10318 raft_consensus.cc:2804] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:19.658123 10002 ts_tablet_manager.cc:1916] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:19.658489 10318 raft_consensus.cc:697] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:19.657960 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet eb7f58645c76441589368072274ebc8f (table default.Table_d5cc86d81fb043319576af0a39c643a8 [id=7af4adcf5ee44bad89afa2e4ded416fc]) successfully deleted
I20260504 13:56:19.664927 9931 tablet_bootstrap.cc:492] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:19.665321 9931 ts_tablet_manager.cc:1403] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.008s
I20260504 13:56:19.668566 9931 raft_consensus.cc:359] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.669271 9931 raft_consensus.cc:385] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.669504 9931 raft_consensus.cc:740] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.670557 10318 consensus_queue.cc:237] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.671387 9931 consensus_queue.cc:260] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.682224 9931 ts_tablet_manager.cc:1434] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260504 13:56:19.683007 9931 tablet_bootstrap.cc:492] T 03fe4ce4f52f48b7b7a804c269e0fd4e P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:19.685930 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b50573310a04f3694eeab2fef0fb540" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
W20260504 13:56:19.686291 9328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20260504 13:56:19.686491 9685 raft_consensus.cc:3060] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.693305 9685 raft_consensus.cc:2468] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:19.664011 10345 raft_consensus.cc:493] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:19.655707 10317 consensus_queue.cc:237] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.697736 10345 raft_consensus.cc:515] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.662935 10013 tablet_bootstrap.cc:492] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:19.699941 10013 ts_tablet_manager.cc:1403] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.115s user 0.010s sys 0.000s
I20260504 13:56:19.701452 9931 tablet_bootstrap.cc:654] T 03fe4ce4f52f48b7b7a804c269e0fd4e P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.702397 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be795f04dd54cf19869c937b91d0ddf" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:19.708104 10319 raft_consensus.cc:515] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.709065 10003 tablet_replica.cc:333] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:19.709822 10003 raft_consensus.cc:2243] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:19.710350 10003 raft_consensus.cc:2272] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:19.713039 10003 ts_tablet_manager.cc:1916] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:19.721328 9686 raft_consensus.cc:2468] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:19.725741 9340 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_6546f8c7ece54e92b8f96bd957ae5851" } new_table_name: "default.Table_0d2ce6c9ff35497b82dd58da3f913ecb" modify_external_catalogs: true
I20260504 13:56:19.733301 9932 tablet_bootstrap.cc:492] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:19.733789 9932 ts_tablet_manager.cc:1403] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.237s user 0.010s sys 0.000s
I20260504 13:56:19.736140 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 91) took 1866 ms. Trace:
I20260504 13:56:19.736521 9944 rpcz_store.cc:276] 0504 13:56:17.870082 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.870176 (+ 94us) service_pool.cc:224] Handling call
0504 13:56:19.341793 (+1471617us) ts_tablet_manager.cc:1151] Deleting tablet 0867df9ea115418193b6d9d3fc251923
0504 13:56:19.341830 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:19.508928 (+167098us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:19.633011 (+124083us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:19.736082 (+103071us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1470631}
I20260504 13:56:19.737998 9944 tablet_replica.cc:333] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
W20260504 13:56:19.744967 9620 leader_election.cc:343] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.745446 9620 leader_election.cc:304] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.746187 10273 raft_consensus.cc:2749] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:19.752732 10319 leader_election.cc:290] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.715827 10013 raft_consensus.cc:359] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.753245 10013 raft_consensus.cc:385] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.753515 10013 raft_consensus.cc:740] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.754050 10013 consensus_queue.cc:260] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.755028 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e4239942f34fb081a026a35740e465" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:19.755421 10013 ts_tablet_manager.cc:1434] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.055s user 0.006s sys 0.000s
I20260504 13:56:19.756074 9686 raft_consensus.cc:2468] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:19.756322 10013 tablet_bootstrap.cc:492] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:19.748862 9931 tablet_bootstrap.cc:492] T 03fe4ce4f52f48b7b7a804c269e0fd4e P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:19.757534 9931 ts_tablet_manager.cc:1403] T 03fe4ce4f52f48b7b7a804c269e0fd4e P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.003s
I20260504 13:56:19.759826 9931 raft_consensus.cc:359] T 03fe4ce4f52f48b7b7a804c269e0fd4e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.722652 9473 leader_election.cc:304] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.761341 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_5ff6dab0e47d41e681533fca02a36153"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:19.762799 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ff6dab0e47d41e681533fca02a36153 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:19.763289 9473 leader_election.cc:304] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:19.764422 10273 raft_consensus.cc:493] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:19.764879 10273 raft_consensus.cc:515] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:19.766889 10013 tablet_bootstrap.cc:654] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.767059 10273 leader_election.cc:290] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:19.768076 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd5f7d3f7c545d18fe19b64e4bc1de6" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:19.768923 10355 raft_consensus.cc:2804] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260504 13:56:19.769140 9618 leader_election.cc:343] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.769544 10355 raft_consensus.cc:493] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:19.771534 10317 raft_consensus.cc:2804] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:19.771796 10355 raft_consensus.cc:3060] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.772143 10317 raft_consensus.cc:697] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:19.773573 9944 raft_consensus.cc:2243] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:19.774022 9944 raft_consensus.cc:2272] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:19.775666 9944 ts_tablet_manager.cc:1916] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:19.776515 9473 leader_election.cc:304] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:19.777863 10317 consensus_queue.cc:237] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.779412 10358 raft_consensus.cc:2804] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:19.779716 10358 raft_consensus.cc:493] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:19.779982 10358 raft_consensus.cc:3060] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.783625 9336 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_1d9e75eee18c499cafb96717ac8bd724" } new_table_name: "default.Table_13d4530198b743ab8d9c025de85e3ff0" modify_external_catalogs: true
I20260504 13:56:19.785022 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c43ff1d759a40d68344f25f066416ac" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:19.737668 10002 ts_tablet_manager.cc:1929] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:19.787438 10002 log.cc:1199] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/afee197774db411d97468913a207703a
I20260504 13:56:19.788121 10002 ts_tablet_manager.cc:1950] T afee197774db411d97468913a207703a P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:19.761504 9932 raft_consensus.cc:359] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:19.791549 9932 raft_consensus.cc:385] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260504 13:56:19.745514 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 99324.8us
I20260504 13:56:19.791885 9932 raft_consensus.cc:740] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.699965 10345 leader_election.cc:290] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.799289 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet afee197774db411d97468913a207703a (table default.Table_91c2d3ed5c5c42758c87f1f726af88c0 [id=a2697dba5f38435c81dc52316712917d]) successfully deleted
I20260504 13:56:19.800034 10358 raft_consensus.cc:515] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:19.808300 9473 leader_election.cc:343] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Illegal state: Tablet not RUNNING: STOPPED
I20260504 13:56:19.809037 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e4239942f34fb081a026a35740e465" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:19.809533 9685 raft_consensus.cc:3060] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.814044 9341 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_434bdc9867ad475a9719ba0c3760ef71" } modify_external_catalogs: true
I20260504 13:56:19.814672 9341 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_434bdc9867ad475a9719ba0c3760ef71" } modify_external_catalogs: true
I20260504 13:56:19.819497 9685 raft_consensus.cc:2468] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
W20260504 13:56:19.826264 9340 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 930078a10afa4dd987ad021cf9609514 has no consensus state
I20260504 13:56:19.774279 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd5f7d3f7c545d18fe19b64e4bc1de6" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:19.827801 9931 raft_consensus.cc:385] T 03fe4ce4f52f48b7b7a804c269e0fd4e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.828085 9931 raft_consensus.cc:740] T 03fe4ce4f52f48b7b7a804c269e0fd4e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.793488 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b50573310a04f3694eeab2fef0fb540" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:19.828794 9834 raft_consensus.cc:3060] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.828696 9931 consensus_queue.cc:260] T 03fe4ce4f52f48b7b7a804c269e0fd4e P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.795037 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e4239942f34fb081a026a35740e465" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:19.796797 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c43ff1d759a40d68344f25f066416ac" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:19.831755 9829 raft_consensus.cc:3060] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.792495 9932 consensus_queue.cc:260] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:19.832903 9931 ts_tablet_manager.cc:1434] T 03fe4ce4f52f48b7b7a804c269e0fd4e P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.075s user 0.005s sys 0.000s
I20260504 13:56:19.802455 10358 leader_election.cc:290] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.810870 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e4239942f34fb081a026a35740e465" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:19.840466 9932 ts_tablet_manager.cc:1434] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.106s user 0.005s sys 0.000s
W20260504 13:56:19.829816 9620 leader_election.cc:343] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.842852 9620 leader_election.cc:304] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:19.846802 9829 raft_consensus.cc:2468] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:19.847333 10213 raft_consensus.cc:2749] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:19.848459 9834 raft_consensus.cc:2468] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:19.850076 9932 tablet_bootstrap.cc:492] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:19.856451 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_073a6dae9da444bea1f96b2f33fe1a85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:19.856731 10358 raft_consensus.cc:493] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:19.861732 10358 raft_consensus.cc:515] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:19.864009 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_073a6dae9da444bea1f96b2f33fe1a85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:19.865586 10358 leader_election.cc:290] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.859051 9473 leader_election.cc:304] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:19.867481 10318 raft_consensus.cc:2804] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
W20260504 13:56:19.874404 9474 leader_election.cc:343] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.876477 10358 consensus_queue.cc:1048] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:19.880935 9474 leader_election.cc:343] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.881325 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1501455a0af348e38746d100987afdd0" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:19.881862 9686 raft_consensus.cc:2468] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:19.882799 9473 leader_election.cc:304] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:19.883795 10358 raft_consensus.cc:2804] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:19.884220 10358 raft_consensus.cc:493] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:19.884555 10358 raft_consensus.cc:3060] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.889062 10013 tablet_bootstrap.cc:492] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:19.889437 10013 ts_tablet_manager.cc:1403] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.133s user 0.008s sys 0.009s
I20260504 13:56:19.881745 9474 leader_election.cc:304] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:19.894641 9932 tablet_bootstrap.cc:654] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.896901 9931 tablet_bootstrap.cc:492] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:19.896863 10345 raft_consensus.cc:2804] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:19.897274 10345 raft_consensus.cc:697] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:19.898213 10345 consensus_queue.cc:237] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.909967 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1501455a0af348e38746d100987afdd0" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:19.911669 9336 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 has no consensus state
I20260504 13:56:19.918591 10318 raft_consensus.cc:697] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:19.919407 10318 consensus_queue.cc:237] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.925393 10358 raft_consensus.cc:515] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.937463 9944 ts_tablet_manager.cc:1929] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:19.937813 9944 log.cc:1199] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/3c43ff1d759a40d68344f25f066416ac
I20260504 13:56:19.938411 9944 ts_tablet_manager.cc:1950] T 3c43ff1d759a40d68344f25f066416ac P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:19.938110 9336 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_18f9a8030cea4cc0abedb81cd5accc59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:19.939828 9336 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18f9a8030cea4cc0abedb81cd5accc59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:19.943712 9474 leader_election.cc:343] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:19.943813 10369 consensus_queue.cc:1048] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:19.949357 10336 raft_consensus.cc:493] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:19.949796 10336 raft_consensus.cc:515] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:19.951790 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 93) took 2080 ms. Trace:
I20260504 13:56:19.952018 10336 leader_election.cc:290] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:19.952102 9944 rpcz_store.cc:276] 0504 13:56:17.870919 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.871380 (+ 461us) service_pool.cc:224] Handling call
0504 13:56:19.737840 (+1866460us) ts_tablet_manager.cc:1151] Deleting tablet 3c43ff1d759a40d68344f25f066416ac
0504 13:56:19.737884 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:19.870300 (+132416us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:19.937349 (+ 67049us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:19.951738 (+ 14389us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1920,"tablet-delete.queue_time_us":1865940}
I20260504 13:56:19.953322 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 92) took 2082 ms. Trace:
I20260504 13:56:19.953832 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7ab3b577b74a9c9a40cb15c94f8aa1" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:19.953655 9944 rpcz_store.cc:276] 0504 13:56:17.870594 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:17.887795 (+ 17201us) service_pool.cc:224] Handling call
0504 13:56:19.953054 (+2065259us) ts_tablet_manager.cc:1151] Deleting tablet 0b7a71dbcc694e1a97019188cfb2c672
0504 13:56:19.953101 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:19.953266 (+ 165us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2064693}
I20260504 13:56:19.954576 9685 raft_consensus.cc:2393] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:19.954699 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 98) took 1513 ms. Trace:
I20260504 13:56:19.958722 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 3c43ff1d759a40d68344f25f066416ac (table default.Table_1f5a66e30f35482dab7c556daebc87c9 [id=c268fa33d24540b4978e3feebb0f793f]) successfully deleted
W20260504 13:56:19.959697 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 0b7a71dbcc694e1a97019188cfb2c672 because the tablet was not found. No further retry: Not found: Tablet not found: 0b7a71dbcc694e1a97019188cfb2c672
I20260504 13:56:19.954996 9944 rpcz_store.cc:276] 0504 13:56:18.441782 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.441878 (+ 96us) service_pool.cc:224] Handling call
0504 13:56:19.954502 (+1512624us) ts_tablet_manager.cc:1151] Deleting tablet 91371bf848f5465ea2db0c090583382d
0504 13:56:19.954539 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:19.954658 (+ 119us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1512145}
W20260504 13:56:19.960860 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 91371bf848f5465ea2db0c090583382d because the tablet was not found. No further retry: Not found: Tablet not found: 91371bf848f5465ea2db0c090583382d
I20260504 13:56:19.961512 9944 tablet_replica.cc:333] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:19.962249 9944 raft_consensus.cc:2243] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:19.962656 9944 raft_consensus.cc:2272] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:19.962143 10013 raft_consensus.cc:359] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:19.962980 10013 raft_consensus.cc:385] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.963290 10013 raft_consensus.cc:740] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.964291 9944 ts_tablet_manager.cc:1916] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:19.973199 10013 consensus_queue.cc:260] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:19.975972 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1501455a0af348e38746d100987afdd0" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:19.976466 9685 raft_consensus.cc:3060] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:19.987340 10013 ts_tablet_manager.cc:1434] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.098s user 0.006s sys 0.002s
I20260504 13:56:19.987910 9931 tablet_bootstrap.cc:654] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:19.990456 10358 leader_election.cc:290] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:19.983064 9685 raft_consensus.cc:2468] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:19.991820 9473 leader_election.cc:304] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:19.992995 9932 tablet_bootstrap.cc:492] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:19.993155 10358 raft_consensus.cc:2804] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:19.993484 9932 ts_tablet_manager.cc:1403] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.152s user 0.021s sys 0.004s
I20260504 13:56:19.995692 9932 raft_consensus.cc:359] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:19.996345 9932 raft_consensus.cc:385] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:19.997006 9932 raft_consensus.cc:740] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:19.994275 10358 raft_consensus.cc:697] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:19.997673 9932 consensus_queue.cc:260] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:19.999200 10013 tablet_bootstrap.cc:492] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:20.000223 9932 ts_tablet_manager.cc:1434] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260504 13:56:20.001359 9932 tablet_bootstrap.cc:492] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:19.998250 10358 consensus_queue.cc:237] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.006470 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_0134ffe7086440ccbcadd076d8fe5d01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:20.008044 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0134ffe7086440ccbcadd076d8fe5d01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:20.008490 9944 ts_tablet_manager.cc:1929] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:20.008824 9944 log.cc:1199] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/a43cdb767fd14ea09b4db4d2881b69aa
I20260504 13:56:20.009421 9944 ts_tablet_manager.cc:1950] T a43cdb767fd14ea09b4db4d2881b69aa P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
W20260504 13:56:20.013391 9618 consensus_peers.cc:597] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3793fe8f1ec43ad809403c4593f5ef6. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:20.016436 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 97) took 1575 ms. Trace:
I20260504 13:56:20.016729 9944 rpcz_store.cc:276] 0504 13:56:18.441485 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.442136 (+ 651us) service_pool.cc:224] Handling call
0504 13:56:19.961377 (+1519241us) ts_tablet_manager.cc:1151] Deleting tablet a43cdb767fd14ea09b4db4d2881b69aa
0504 13:56:19.961413 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:19.995911 (+ 34498us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:20.008390 (+ 12479us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:20.016382 (+ 7992us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1510340}
I20260504 13:56:20.017699 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet a43cdb767fd14ea09b4db4d2881b69aa (table default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]) successfully deleted
I20260504 13:56:20.019515 9944 tablet_replica.cc:333] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:20.020241 9944 raft_consensus.cc:2243] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:20.020619 9944 raft_consensus.cc:2272] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:20.025731 9944 ts_tablet_manager.cc:1916] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:20.019610 10013 tablet_bootstrap.cc:654] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:20.050377 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1501455a0af348e38746d100987afdd0" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:20.051784 9336 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_7e17be2257534659b742a47c80f3afa6" } new_table_name: "default.Table_91cc6c38c9ff43dca25814664068b1c6" modify_external_catalogs: true
I20260504 13:56:20.051831 10358 raft_consensus.cc:493] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:20.052393 10358 raft_consensus.cc:515] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.056214 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70caac2e256d419dbc2e70cb5dfffe97" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:20.056779 9685 raft_consensus.cc:2468] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:20.057118 10355 raft_consensus.cc:515] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.057741 9473 leader_election.cc:304] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:20.064872 10318 raft_consensus.cc:2804] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:20.065197 10318 raft_consensus.cc:493] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:20.065497 10318 raft_consensus.cc:3060] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.066689 10355 leader_election.cc:290] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:20.069132 9944 ts_tablet_manager.cc:1929] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:20.069478 9944 log.cc:1199] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/14b7f1e834d0439c89abb180bd13853d
I20260504 13:56:20.070056 9944 ts_tablet_manager.cc:1950] T 14b7f1e834d0439c89abb180bd13853d P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:20.072238 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a428049fdb4a68aae729400e2d9674" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:20.074358 9685 raft_consensus.cc:3060] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.074906 9931 tablet_bootstrap.cc:492] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:20.077307 9931 ts_tablet_manager.cc:1403] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.181s user 0.003s sys 0.008s
I20260504 13:56:20.059641 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe7ab3b577b74a9c9a40cb15c94f8aa1" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:20.058511 9474 leader_election.cc:343] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:20.085232 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 99) took 1643 ms. Trace:
I20260504 13:56:20.085541 9944 rpcz_store.cc:276] 0504 13:56:18.442054 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.453783 (+ 11729us) service_pool.cc:224] Handling call
0504 13:56:20.019382 (+1565599us) ts_tablet_manager.cc:1151] Deleting tablet 14b7f1e834d0439c89abb180bd13853d
0504 13:56:20.019419 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:20.042648 (+ 23229us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:20.069014 (+ 26366us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:20.085017 (+ 16003us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1565127}
I20260504 13:56:20.086525 9944 tablet_replica.cc:333] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:20.087312 9944 raft_consensus.cc:2243] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:20.087723 9944 raft_consensus.cc:2272] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:20.088007 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70caac2e256d419dbc2e70cb5dfffe97" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:20.089334 9944 ts_tablet_manager.cc:1916] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:20.089308 9767 leader_election.cc:304] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:20.090257 10336 raft_consensus.cc:3060] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.090271 9931 raft_consensus.cc:359] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.090941 9931 raft_consensus.cc:385] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:20.091027 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 14b7f1e834d0439c89abb180bd13853d (table default.Table_a0a1002687744b879e47ae4fd701d903 [id=2d45d703594b416893908650027e94b3]) successfully deleted
I20260504 13:56:20.091244 9931 raft_consensus.cc:740] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:20.093802 9931 consensus_queue.cc:260] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.079380 10358 leader_election.cc:290] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:20.095660 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07a428049fdb4a68aae729400e2d9674" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:20.096203 9834 raft_consensus.cc:3060] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.097344 10318 raft_consensus.cc:515] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.099792 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70caac2e256d419dbc2e70cb5dfffe97" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:20.099946 9685 raft_consensus.cc:2468] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:20.100312 9686 raft_consensus.cc:3060] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.101051 9473 leader_election.cc:304] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:20.101866 10358 raft_consensus.cc:2804] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:20.104765 10358 raft_consensus.cc:697] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:20.105967 10358 consensus_queue.cc:237] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.107627 9686 raft_consensus.cc:2468] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:20.108628 9473 leader_election.cc:304] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:20.109334 10345 raft_consensus.cc:2804] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:20.109741 10345 raft_consensus.cc:697] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:20.110427 10345 consensus_queue.cc:237] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.112421 10336 raft_consensus.cc:2749] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:20.116979 9474 leader_election.cc:343] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:20.119159 10317 raft_consensus.cc:493] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:20.119603 10317 raft_consensus.cc:515] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.121337 10317 leader_election.cc:290] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:20.122653 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03fe4ce4f52f48b7b7a804c269e0fd4e" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:20.123266 9686 raft_consensus.cc:2468] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:20.124176 9473 leader_election.cc:304] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:20.024184 10003 ts_tablet_manager.cc:1929] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:20.129065 10003 log.cc:1199] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/69ed77782cd34274932957106ce26766
I20260504 13:56:20.130936 10003 ts_tablet_manager.cc:1950] T 69ed77782cd34274932957106ce26766 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:20.138029 9944 ts_tablet_manager.cc:1929] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:20.138377 9944 log.cc:1199] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/69ed77782cd34274932957106ce26766
I20260504 13:56:20.126504 10318 leader_election.cc:290] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:20.138916 9944 ts_tablet_manager.cc:1950] T 69ed77782cd34274932957106ce26766 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:20.139159 10355 raft_consensus.cc:2804] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:20.140631 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 100) took 1686 ms. Trace:
I20260504 13:56:20.139813 10355 raft_consensus.cc:493] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:20.141287 10355 raft_consensus.cc:3060] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.140934 9944 rpcz_store.cc:276] 0504 13:56:18.454656 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.455284 (+ 628us) service_pool.cc:224] Handling call
0504 13:56:20.086393 (+1631109us) ts_tablet_manager.cc:1151] Deleting tablet 69ed77782cd34274932957106ce26766
0504 13:56:20.086430 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:20.119204 (+ 32774us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:20.137915 (+ 18711us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:20.140581 (+ 2666us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1630617}
I20260504 13:56:20.142231 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 69ed77782cd34274932957106ce26766 (table default.Table_cdaaea60f9354320bc4cfee6729ba12f [id=55db3ce6e7694cca94a1705fb1c0e90e]) successfully deleted
I20260504 13:56:20.144817 10013 tablet_bootstrap.cc:492] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:20.145220 10013 ts_tablet_manager.cc:1403] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.146s user 0.008s sys 0.009s
I20260504 13:56:20.148131 10013 raft_consensus.cc:359] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.148658 10013 raft_consensus.cc:385] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:20.148864 10013 raft_consensus.cc:740] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:20.149379 10013 consensus_queue.cc:260] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.155319 10355 raft_consensus.cc:515] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.162823 9344 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet b8e4239942f34fb081a026a35740e465 (table default.Table_11d8ab61a6044154bb599875a0bc9004 [id=73e12524eb9e416782bf9f7fb58c93bd]). Expected version 1 got 0
I20260504 13:56:20.163647 9344 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:20.164333 9344 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 48b193e958b4414aad29dd2bcee1a61b (table default.Table_2c39b14515bb46f6a0b0f1252d8e273d [id=e9e56499fd2243919308927b3614ae8e]). Expected version 1 got 0
I20260504 13:56:20.156118 10013 ts_tablet_manager.cc:1434] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.010s user 0.001s sys 0.004s
I20260504 13:56:20.163007 9515 tablet_service.cc:1511] Processing CreateTablet for tablet 930078a10afa4dd987ad021cf9609514 (DEFAULT_TABLE table=default.Table_0d2ce6c9ff35497b82dd58da3f913ecb [id=5a0c5354346746c186e36fe23406442c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.168236 10013 tablet_bootstrap.cc:492] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:20.168500 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 930078a10afa4dd987ad021cf9609514. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.172302 10355 leader_election.cc:290] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:20.172806 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03fe4ce4f52f48b7b7a804c269e0fd4e" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:20.173679 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03fe4ce4f52f48b7b7a804c269e0fd4e" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:20.173898 9829 raft_consensus.cc:2468] T 03fe4ce4f52f48b7b7a804c269e0fd4e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:20.166790 9518 tablet_service.cc:1511] Processing CreateTablet for tablet 5254b1061e2b424487f5d309baaee6b6 (DEFAULT_TABLE table=default.Table_798d2540b8f4489fb11763ccaa36bced [id=049b4fa786fc4dbb8e52931dcaaab90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.174278 9686 raft_consensus.cc:3060] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.174855 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5254b1061e2b424487f5d309baaee6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.181594 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70caac2e256d419dbc2e70cb5dfffe97" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:20.182052 9338 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:55533 (request call id 37) took 871 ms (client timeout 1000 ms). Trace:
I20260504 13:56:20.182392 9686 raft_consensus.cc:2468] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
W20260504 13:56:20.182348 9338 rpcz_store.cc:269] 0504 13:56:19.310746 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:19.322411 (+ 11665us) service_pool.cc:224] Handling call
0504 13:56:19.437161 (+114750us) spinlock_profiling.cc:243] Waited 84.7 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:20.181982 (+744821us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":194777344}
W20260504 13:56:20.183351 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 63120.6us
I20260504 13:56:20.183923 10013 tablet_bootstrap.cc:654] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:20.185333 9810 tablet_service.cc:1511] Processing CreateTablet for tablet 930078a10afa4dd987ad021cf9609514 (DEFAULT_TABLE table=default.Table_0d2ce6c9ff35497b82dd58da3f913ecb [id=5a0c5354346746c186e36fe23406442c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.183650 9665 tablet_service.cc:1511] Processing CreateTablet for tablet 930078a10afa4dd987ad021cf9609514 (DEFAULT_TABLE table=default.Table_0d2ce6c9ff35497b82dd58da3f913ecb [id=5a0c5354346746c186e36fe23406442c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.186412 9810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 930078a10afa4dd987ad021cf9609514. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.186677 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 930078a10afa4dd987ad021cf9609514. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.192238 9808 tablet_service.cc:1511] Processing CreateTablet for tablet 5254b1061e2b424487f5d309baaee6b6 (DEFAULT_TABLE table=default.Table_798d2540b8f4489fb11763ccaa36bced [id=049b4fa786fc4dbb8e52931dcaaab90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.193245 9808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5254b1061e2b424487f5d309baaee6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.195900 9473 leader_election.cc:304] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:20.198900 10345 raft_consensus.cc:2804] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:20.199347 10345 raft_consensus.cc:697] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:20.200126 10317 raft_consensus.cc:493] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:20.200551 10317 raft_consensus.cc:515] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.202044 9663 tablet_service.cc:1511] Processing CreateTablet for tablet 5254b1061e2b424487f5d309baaee6b6 (DEFAULT_TABLE table=default.Table_798d2540b8f4489fb11763ccaa36bced [id=049b4fa786fc4dbb8e52931dcaaab90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.202282 10317 leader_election.cc:290] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:20.203023 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5254b1061e2b424487f5d309baaee6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.200148 10345 consensus_queue.cc:237] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.207116 9660 tablet_service.cc:1511] Processing CreateTablet for tablet c3112c3c3629412e9b76407c7cba8f92 (DEFAULT_TABLE table=default.Table_b59e5e285d50401e94d51afeb4cbc500 [id=8f16af9aec63405cbae2a91b84fa3009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.207829 9513 tablet_service.cc:1511] Processing CreateTablet for tablet c3112c3c3629412e9b76407c7cba8f92 (DEFAULT_TABLE table=default.Table_b59e5e285d50401e94d51afeb4cbc500 [id=8f16af9aec63405cbae2a91b84fa3009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.208129 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3112c3c3629412e9b76407c7cba8f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.208719 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3112c3c3629412e9b76407c7cba8f92. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:20.209686 9474 leader_election.cc:343] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:20.211786 10003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 111) took 1817 ms. Trace:
I20260504 13:56:20.212107 10003 rpcz_store.cc:276] 0504 13:56:18.394581 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.394692 (+ 111us) service_pool.cc:224] Handling call
0504 13:56:19.708914 (+1314222us) ts_tablet_manager.cc:1151] Deleting tablet 69ed77782cd34274932957106ce26766
0504 13:56:19.708968 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:19.926248 (+217280us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:20.024065 (+ 97817us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:20.211510 (+187445us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290060}
I20260504 13:56:20.128655 9932 tablet_bootstrap.cc:654] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:20.195067 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet b8e4239942f34fb081a026a35740e465 (table default.Table_11d8ab61a6044154bb599875a0bc9004 [id=73e12524eb9e416782bf9f7fb58c93bd]). Expected version 1 got 0
I20260504 13:56:20.217257 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 0be795f04dd54cf19869c937b91d0ddf (table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]). Expected version 1 got 0
I20260504 13:56:20.217803 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:20.218554 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 48b193e958b4414aad29dd2bcee1a61b (table default.Table_2c39b14515bb46f6a0b0f1252d8e273d [id=e9e56499fd2243919308927b3614ae8e]). Expected version 1 got 0
W20260504 13:56:20.219929 9330 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 has no consensus state
I20260504 13:56:20.219920 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be795f04dd54cf19869c937b91d0ddf" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:20.219378 9337 catalog_manager.cc:5671] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } }
I20260504 13:56:20.220597 9686 raft_consensus.cc:2468] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:20.221150 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]). Expected version 1 got 0
I20260504 13:56:20.221668 9473 leader_election.cc:304] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:20.225008 10318 raft_consensus.cc:2804] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:20.225327 10318 raft_consensus.cc:493] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:20.225597 10318 raft_consensus.cc:3060] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.227447 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 69ed77782cd34274932957106ce26766 (table default.Table_cdaaea60f9354320bc4cfee6729ba12f [id=55db3ce6e7694cca94a1705fb1c0e90e]) successfully deleted
I20260504 13:56:20.228600 9813 tablet_service.cc:1558] Processing DeleteTablet for tablet cdb86f6370bf4c5f9c488d85efb67ef7 with delete_type TABLET_DATA_DELETED (replaced by tablet 030b35c3409143108c34ab1f0ee1b96d) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:20.229423 9812 tablet_service.cc:1511] Processing CreateTablet for tablet c3112c3c3629412e9b76407c7cba8f92 (DEFAULT_TABLE table=default.Table_b59e5e285d50401e94d51afeb4cbc500 [id=8f16af9aec63405cbae2a91b84fa3009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.230427 9812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3112c3c3629412e9b76407c7cba8f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.232479 10318 raft_consensus.cc:515] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.233340 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_225ad80b8318432b88a729064c88e8ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:20.234565 10318 leader_election.cc:290] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:20.241771 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be795f04dd54cf19869c937b91d0ddf" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:20.242449 9686 raft_consensus.cc:3060] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.248750 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03fe4ce4f52f48b7b7a804c269e0fd4e" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:20.249257 9829 raft_consensus.cc:3060] T 03fe4ce4f52f48b7b7a804c269e0fd4e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.256736 9829 raft_consensus.cc:2468] T 03fe4ce4f52f48b7b7a804c269e0fd4e P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:20.259881 9808 tablet_service.cc:1511] Processing CreateTablet for tablet 7cb6927537704cc4b17617cbdaa21ac1 (DEFAULT_TABLE table=default.Table_03e728dbf91f4fd19afe882c3b4d7bda [id=bab1a941deda44fabc9bf67ce6ab0d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.261431 9808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cb6927537704cc4b17617cbdaa21ac1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:20.263339 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_225ad80b8318432b88a729064c88e8ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:20.262166 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be795f04dd54cf19869c937b91d0ddf" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:20.269532 9474 leader_election.cc:343] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:20.270341 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be795f04dd54cf19869c937b91d0ddf" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:20.263154 10003 tablet_replica.cc:333] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:20.272356 10003 raft_consensus.cc:2243] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:20.272850 10003 raft_consensus.cc:2272] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:20.274598 10003 ts_tablet_manager.cc:1916] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:20.281288 9813 tablet_service.cc:1511] Processing CreateTablet for tablet 8db7c4732fa048f9950248f45d144c57 (DEFAULT_TABLE table=default.Table_a8d8f096360b4b5281f5124fe966098a [id=5b5423cfb646459cb57b9429e44efc62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.282166 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8db7c4732fa048f9950248f45d144c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.281980 9665 tablet_service.cc:1511] Processing CreateTablet for tablet 7cb6927537704cc4b17617cbdaa21ac1 (DEFAULT_TABLE table=default.Table_03e728dbf91f4fd19afe882c3b4d7bda [id=bab1a941deda44fabc9bf67ce6ab0d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.290812 9809 tablet_service.cc:1511] Processing CreateTablet for tablet 1fbe09b4078c4eebb52d7d00bc87371c (DEFAULT_TABLE table=default.Table_bdb22fcd488647e0bfed4170cb70365d [id=2560e279f4504b35b3f9309673776eee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.291934 9809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fbe09b4078c4eebb52d7d00bc87371c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.302644 9834 raft_consensus.cc:2468] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:20.320380 9666 tablet_service.cc:1511] Processing CreateTablet for tablet 8db7c4732fa048f9950248f45d144c57 (DEFAULT_TABLE table=default.Table_a8d8f096360b4b5281f5124fe966098a [id=5b5423cfb646459cb57b9429e44efc62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.274473 9513 tablet_service.cc:1511] Processing CreateTablet for tablet 7cb6927537704cc4b17617cbdaa21ac1 (DEFAULT_TABLE table=default.Table_03e728dbf91f4fd19afe882c3b4d7bda [id=bab1a941deda44fabc9bf67ce6ab0d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.275907 10013 tablet_bootstrap.cc:492] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:20.284379 9520 tablet_service.cc:1511] Processing CreateTablet for tablet 8db7c4732fa048f9950248f45d144c57 (DEFAULT_TABLE table=default.Table_a8d8f096360b4b5281f5124fe966098a [id=5b5423cfb646459cb57b9429e44efc62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.321810 10013 ts_tablet_manager.cc:1403] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.154s user 0.005s sys 0.008s
I20260504 13:56:20.322153 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cb6927537704cc4b17617cbdaa21ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.322757 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8db7c4732fa048f9950248f45d144c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.324280 10013 raft_consensus.cc:359] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.324847 10013 raft_consensus.cc:385] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:20.325089 10013 raft_consensus.cc:740] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:20.325771 10013 consensus_queue.cc:260] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.327934 10318 raft_consensus.cc:493] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:20.328133 10013 ts_tablet_manager.cc:1434] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260504 13:56:20.328326 10318 raft_consensus.cc:515] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.329236 10013 tablet_bootstrap.cc:492] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:20.330451 10318 leader_election.cc:290] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:20.333262 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8db7c4732fa048f9950248f45d144c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.336706 9339 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 2b50573310a04f3694eeab2fef0fb540
I20260504 13:56:20.337960 9339 catalog_manager.cc:6741] ReplaceTablet: tablet 2b50573310a04f3694eeab2fef0fb540 deleted and replaced by tablet 8bb77106b48a45a2aaa938dc474fa8bd
I20260504 13:56:20.338235 9516 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b50573310a04f3694eeab2fef0fb540 with delete_type TABLET_DATA_DELETED (replaced by tablet 8bb77106b48a45a2aaa938dc474fa8bd) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:20.338280 9339 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:35768 (request call id 182) took 1469 ms. Trace:
I20260504 13:56:20.339457 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cb6927537704cc4b17617cbdaa21ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.343607 9664 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b50573310a04f3694eeab2fef0fb540 with delete_type TABLET_DATA_DELETED (replaced by tablet 8bb77106b48a45a2aaa938dc474fa8bd) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:20.345371 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34877a81d504f46a2f680c5d5ddb4f4" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:20.345922 9685 raft_consensus.cc:2468] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:20.338526 9339 rpcz_store.cc:276] 0504 13:56:18.869134 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.869233 (+ 99us) service_pool.cc:224] Handling call
0504 13:56:18.890881 (+ 21648us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b3491 00007fc27b2de227 00007fc27b2de1b2 00007fc27b2de0e1 00007fc27b2de05c 00007fc27b2dd312 00007fc27958bbcc 00007fc27958b006
0504 13:56:20.338218 (+1447337us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:20.323683 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:20.324050 (+ 367us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:20.324072 (+ 22us) write_op.cc:435] Acquired schema lock
0504 13:56:20.324084 (+ 12us) tablet.cc:662] Decoding operations
0504 13:56:20.324363 (+ 279us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:20.324400 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:20.324422 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:20.324597 (+ 175us) tablet.cc:701] Row locks acquired
0504 13:56:20.324609 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:20.324699 (+ 90us) write_op.cc:270] Start()
0504 13:56:20.324787 (+ 88us) write_op.cc:276] Timestamp: P: 1777902980324660 usec, L: 0
0504 13:56:20.324802 (+ 15us) op_driver.cc:348] REPLICATION: starting
0504 13:56:20.325241 (+ 439us) log.cc:844] Serialized 628 byte log entry
0504 13:56:20.334195 (+ 8954us) op_driver.cc:464] REPLICATION: finished
0504 13:56:20.334500 (+ 305us) write_op.cc:301] APPLY: starting
0504 13:56:20.334549 (+ 49us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:20.334647 (+ 98us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:20.334746 (+ 99us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:20.335053 (+ 307us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:20.335287 (+ 234us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:20.335336 (+ 49us) write_op.cc:312] APPLY: finished
0504 13:56:20.335505 (+ 169us) log.cc:844] Serialized 29 byte log entry
0504 13:56:20.335635 (+ 130us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:20.335698 (+ 63us) write_op.cc:454] Released schema lock
0504 13:56:20.335730 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":47490432,"child_traces":[["op",{"apply.queue_time_us":185,"apply.run_cpu_time_us":1403,"apply.run_wall_time_us":1403,"num_ops":2,"prepare.queue_time_us":168765,"raft.queue_time_us":68,"raft.run_cpu_time_us":534,"raft.run_wall_time_us":534,"replication_time_us":9319}]]}
I20260504 13:56:20.346894 9473 leader_election.cc:304] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:20.348999 10345 raft_consensus.cc:2804] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:20.349316 10345 raft_consensus.cc:493] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:20.349601 10345 raft_consensus.cc:3060] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.355234 9944 tablet_replica.cc:333] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
W20260504 13:56:20.321970 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 52471.4us
W20260504 13:56:20.357008 9474 leader_election.cc:343] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:20.359690 9339 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_a8d8f096360b4b5281f5124fe966098a" } new_table_name: "default.Table_f8431c2e71824e90844d341601e3be4e" modify_external_catalogs: true
I20260504 13:56:20.367273 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34877a81d504f46a2f680c5d5ddb4f4" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:20.368124 9814 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b50573310a04f3694eeab2fef0fb540 with delete_type TABLET_DATA_DELETED (replaced by tablet 8bb77106b48a45a2aaa938dc474fa8bd) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:20.378055 9931 ts_tablet_manager.cc:1434] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.298s user 0.009s sys 0.013s
W20260504 13:56:20.386639 9474 leader_election.cc:343] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:20.392836 9686 raft_consensus.cc:2468] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:20.394397 10318 consensus_queue.cc:1048] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:20.402878 9932 tablet_bootstrap.cc:492] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:20.403399 9932 ts_tablet_manager.cc:1403] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.402s user 0.012s sys 0.000s
I20260504 13:56:20.405282 10391 tablet_replica.cc:333] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:20.405941 9932 raft_consensus.cc:359] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.406301 10391 raft_consensus.cc:2243] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:20.406526 9932 raft_consensus.cc:385] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:20.406764 9932 raft_consensus.cc:740] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:20.407187 10391 pending_rounds.cc:70] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00: Trying to abort 1 pending ops.
I20260504 13:56:20.408694 9931 tablet_bootstrap.cc:492] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:20.414008 9944 raft_consensus.cc:2243] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:20.434979 10391 pending_rounds.cc:77] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290605246660608 op_type: NO_OP noop_request { }
I20260504 13:56:20.435122 10395 raft_consensus.cc:493] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:20.435490 10391 raft_consensus.cc:2889] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:20.435981 10391 raft_consensus.cc:2272] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:20.425372 9516 tablet_service.cc:1511] Processing CreateTablet for tablet 1fbe09b4078c4eebb52d7d00bc87371c (DEFAULT_TABLE table=default.Table_bdb22fcd488647e0bfed4170cb70365d [id=2560e279f4504b35b3f9309673776eee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.433228 9473 leader_election.cc:304] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:20.436121 10395 raft_consensus.cc:515] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.438757 9342 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 50e129980cd84f6ca22cbf2addb06230
I20260504 13:56:20.439988 9342 catalog_manager.cc:6741] ReplaceTablet: tablet 50e129980cd84f6ca22cbf2addb06230 deleted and replaced by tablet c47b6e3b4d0c45c28384f7aae6b749de
I20260504 13:56:20.440292 9342 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:35768 (request call id 183) took 1496 ms. Trace:
I20260504 13:56:20.440555 9342 rpcz_store.cc:276] 0504 13:56:18.944248 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.944344 (+ 96us) service_pool.cc:224] Handling call
0504 13:56:20.440245 (+1495901us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:20.364873 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:20.365331 (+ 458us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:20.365361 (+ 30us) write_op.cc:435] Acquired schema lock
0504 13:56:20.365380 (+ 19us) tablet.cc:662] Decoding operations
0504 13:56:20.365726 (+ 346us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:20.365775 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:20.365804 (+ 29us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:20.366033 (+ 229us) tablet.cc:701] Row locks acquired
0504 13:56:20.392307 (+ 26274us) write_op.cc:260] PREPARE: finished
0504 13:56:20.392415 (+ 108us) write_op.cc:270] Start()
0504 13:56:20.392496 (+ 81us) write_op.cc:276] Timestamp: P: 1777902980392370 usec, L: 0
0504 13:56:20.392511 (+ 15us) op_driver.cc:348] REPLICATION: starting
0504 13:56:20.392987 (+ 476us) log.cc:844] Serialized 628 byte log entry
0504 13:56:20.436535 (+ 43548us) op_driver.cc:464] REPLICATION: finished
0504 13:56:20.436901 (+ 366us) write_op.cc:301] APPLY: starting
0504 13:56:20.436951 (+ 50us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:20.437058 (+ 107us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:20.437146 (+ 88us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:20.437524 (+ 378us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:20.437736 (+ 212us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:20.437779 (+ 43us) write_op.cc:312] APPLY: finished
0504 13:56:20.437950 (+ 171us) log.cc:844] Serialized 29 byte log entry
0504 13:56:20.438085 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:20.438169 (+ 84us) write_op.cc:454] Released schema lock
0504 13:56:20.438199 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":229,"apply.run_cpu_time_us":1491,"apply.run_wall_time_us":1491,"num_ops":2,"prepare.queue_time_us":204545,"raft.queue_time_us":84,"raft.run_cpu_time_us":554,"raft.run_wall_time_us":553,"replication_time_us":43916}]]}
I20260504 13:56:20.445082 9513 tablet_service.cc:1558] Processing DeleteTablet for tablet 50e129980cd84f6ca22cbf2addb06230 with delete_type TABLET_DATA_DELETED (replaced by tablet c47b6e3b4d0c45c28384f7aae6b749de) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:20.445464 10369 raft_consensus.cc:2804] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:20.445896 10369 raft_consensus.cc:697] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:20.446660 10369 consensus_queue.cc:237] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.449474 9664 tablet_service.cc:1558] Processing DeleteTablet for tablet 50e129980cd84f6ca22cbf2addb06230 with delete_type TABLET_DATA_DELETED (replaced by tablet c47b6e3b4d0c45c28384f7aae6b749de) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:20.450187 9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fbe09b4078c4eebb52d7d00bc87371c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.456213 9808 tablet_service.cc:1558] Processing DeleteTablet for tablet 50e129980cd84f6ca22cbf2addb06230 with delete_type TABLET_DATA_DELETED (replaced by tablet c47b6e3b4d0c45c28384f7aae6b749de) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:20.457676 10013 tablet_bootstrap.cc:654] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:20.457649 9944 raft_consensus.cc:2272] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:20.461606 10345 raft_consensus.cc:515] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.464111 10345 leader_election.cc:290] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:20.464442 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34877a81d504f46a2f680c5d5ddb4f4" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:20.465011 9686 raft_consensus.cc:3060] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.466001 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34877a81d504f46a2f680c5d5ddb4f4" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:20.467784 9474 leader_election.cc:343] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:20.470322 9931 tablet_bootstrap.cc:654] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:20.475798 10395 leader_election.cc:290] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:20.477380 9944 ts_tablet_manager.cc:1916] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:20.477547 9660 tablet_service.cc:1511] Processing CreateTablet for tablet 1fbe09b4078c4eebb52d7d00bc87371c (DEFAULT_TABLE table=default.Table_bdb22fcd488647e0bfed4170cb70365d [id=2560e279f4504b35b3f9309673776eee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.478583 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fbe09b4078c4eebb52d7d00bc87371c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.479384 9932 consensus_queue.cc:260] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.481156 9932 ts_tablet_manager.cc:1434] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.077s user 0.004s sys 0.000s
I20260504 13:56:20.482010 9932 tablet_bootstrap.cc:492] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:20.482649 10391 ts_tablet_manager.cc:1916] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:20.487521 9341 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet c34630d55f4a4e19b33ddf4c13995a5b
I20260504 13:56:20.496673 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_2d5a582adb634895a55193dd861f3b60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:20.498229 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d5a582adb634895a55193dd861f3b60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:20.500695 9520 tablet_service.cc:1558] Processing DeleteTablet for tablet c34630d55f4a4e19b33ddf4c13995a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:19 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:20.504249 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd8cd1caaeb249e3bafee7f1f0d4df2f" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:20.505970 9539 raft_consensus.cc:2468] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:20.506217 10355 consensus_queue.cc:1048] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:20.517765 9663 tablet_service.cc:1558] Processing DeleteTablet for tablet c34630d55f4a4e19b33ddf4c13995a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:19 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:20.522025 9686 raft_consensus.cc:2468] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:20.522449 9520 tablet_service.cc:1511] Processing CreateTablet for tablet c8989cdde4ca4e349079596e33723b2c (DEFAULT_TABLE table=default.Table_2d928cb3553f40bc8e825d0741ef3199 [id=51b254be98ee457d892039784e4a5f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.523422 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8989cdde4ca4e349079596e33723b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.547850 9473 leader_election.cc:304] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:20.548790 10013 tablet_bootstrap.cc:492] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:20.549140 10013 ts_tablet_manager.cc:1403] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.220s user 0.009s sys 0.000s
I20260504 13:56:20.551277 10013 raft_consensus.cc:359] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.551823 10013 raft_consensus.cc:385] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:20.552050 10013 raft_consensus.cc:740] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:20.552554 10013 consensus_queue.cc:260] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.553371 10369 raft_consensus.cc:2804] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:20.553704 9932 tablet_bootstrap.cc:654] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:20.553774 10369 raft_consensus.cc:697] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:20.554351 10013 ts_tablet_manager.cc:1434] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260504 13:56:20.554466 10369 consensus_queue.cc:237] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.526014 9618 leader_election.cc:304] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:20.537748 9663 tablet_service.cc:1511] Processing CreateTablet for tablet c8989cdde4ca4e349079596e33723b2c (DEFAULT_TABLE table=default.Table_2d928cb3553f40bc8e825d0741ef3199 [id=51b254be98ee457d892039784e4a5f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.559979 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8989cdde4ca4e349079596e33723b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.561269 10013 tablet_bootstrap.cc:492] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:20.477053 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd8cd1caaeb249e3bafee7f1f0d4df2f" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:20.567626 9620 leader_election.cc:343] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:20.574265 10013 tablet_bootstrap.cc:654] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:20.576040 10393 raft_consensus.cc:2804] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:20.576388 10393 raft_consensus.cc:493] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:20.577231 10393 raft_consensus.cc:3060] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.578415 9808 tablet_service.cc:1511] Processing CreateTablet for tablet 9f3905bfb30e4b03a7fcf0a2f57e2117 (DEFAULT_TABLE table=default.Table_597b4ca3e37849d0aa9c86eed64d0614 [id=5a597e1cae1d4126ae355289aef1ced6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.579419 9808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f3905bfb30e4b03a7fcf0a2f57e2117. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:20.579555 9618 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 53618us
I20260504 13:56:20.582032 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 48b193e958b4414aad29dd2bcee1a61b (table default.Table_2c39b14515bb46f6a0b0f1252d8e273d [id=e9e56499fd2243919308927b3614ae8e]). Expected version 1 got 0
I20260504 13:56:20.582690 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]). Expected version 1 got 0
I20260504 13:56:20.586748 10355 consensus_queue.cc:1048] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:20.592265 9338 catalog_manager.cc:5671] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:20.607370 10318 consensus_queue.cc:1048] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:20.607905 10393 raft_consensus.cc:515] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.616050 9932 tablet_bootstrap.cc:492] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:20.616286 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd8cd1caaeb249e3bafee7f1f0d4df2f" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:20.617648 9620 leader_election.cc:343] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:20.618611 9932 ts_tablet_manager.cc:1403] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.137s user 0.012s sys 0.000s
I20260504 13:56:20.619112 10395 raft_consensus.cc:493] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:20.619593 10395 raft_consensus.cc:515] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.620770 10003 ts_tablet_manager.cc:1929] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:20.621737 10393 leader_election.cc:290] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:20.623186 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_a5fa0a0bead94aa3b02d6175fefde171"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:20.624650 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5fa0a0bead94aa3b02d6175fefde171 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:20.624624 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd8cd1caaeb249e3bafee7f1f0d4df2f" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:20.625113 9539 raft_consensus.cc:3060] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.628746 9932 raft_consensus.cc:359] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.629518 9932 raft_consensus.cc:385] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:20.634152 10395 leader_election.cc:290] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:20.635944 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34630d55f4a4e19b33ddf4c13995a5b" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:20.639302 9618 leader_election.cc:343] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:20.629804 9932 raft_consensus.cc:740] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:20.641093 9932 consensus_queue.cc:260] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.643270 9518 tablet_service.cc:1511] Processing CreateTablet for tablet 9f3905bfb30e4b03a7fcf0a2f57e2117 (DEFAULT_TABLE table=default.Table_597b4ca3e37849d0aa9c86eed64d0614 [id=5a597e1cae1d4126ae355289aef1ced6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.644882 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f3905bfb30e4b03a7fcf0a2f57e2117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.643499 9932 ts_tablet_manager.cc:1434] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260504 13:56:20.658543 9932 tablet_bootstrap.cc:492] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:20.668313 10399 log.cc:927] Time spent T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00: Append to log took a long time: real 0.094s user 0.001s sys 0.000s
I20260504 13:56:20.593948 9338 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]). Expected version 1 got 0
I20260504 13:56:20.673334 9338 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:20.674628 9338 catalog_manager.cc:5671] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:20.531875 9812 tablet_service.cc:1558] Processing DeleteTablet for tablet c34630d55f4a4e19b33ddf4c13995a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:19 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:20.679210 9932 tablet_bootstrap.cc:654] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:20.529191 9809 tablet_service.cc:1511] Processing CreateTablet for tablet c8989cdde4ca4e349079596e33723b2c (DEFAULT_TABLE table=default.Table_2d928cb3553f40bc8e825d0741ef3199 [id=51b254be98ee457d892039784e4a5f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.627676 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34630d55f4a4e19b33ddf4c13995a5b" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:20.680866 9809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8989cdde4ca4e349079596e33723b2c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:20.680819 9620 leader_election.cc:343] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:20.681308 9620 leader_election.cc:304] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:20.682512 10395 raft_consensus.cc:2749] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:20.690766 9339 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8db7c4732fa048f9950248f45d144c57 consensus state has no leader
I20260504 13:56:20.698083 9932 tablet_bootstrap.cc:492] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:20.697949 10369 raft_consensus.cc:493] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:20.698463 9932 ts_tablet_manager.cc:1403] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.000s
I20260504 13:56:20.698558 10369 raft_consensus.cc:515] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.701032 10369 leader_election.cc:290] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:20.679350 10003 log.cc:1199] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/afee197774db411d97468913a207703a
I20260504 13:56:20.702104 10003 ts_tablet_manager.cc:1950] T afee197774db411d97468913a207703a P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:20.704391 9663 tablet_service.cc:1511] Processing CreateTablet for tablet 9f3905bfb30e4b03a7fcf0a2f57e2117 (DEFAULT_TABLE table=default.Table_597b4ca3e37849d0aa9c86eed64d0614 [id=5a597e1cae1d4126ae355289aef1ced6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.705523 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f3905bfb30e4b03a7fcf0a2f57e2117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.705775 10003 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 112) took 1548 ms. Trace:
I20260504 13:56:20.706125 10003 rpcz_store.cc:276] 0504 13:56:19.157402 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:19.157512 (+ 110us) service_pool.cc:224] Handling call
0504 13:56:20.262855 (+1105343us) ts_tablet_manager.cc:1151] Deleting tablet afee197774db411d97468913a207703a
0504 13:56:20.262922 (+ 67us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:20.506655 (+243733us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:20.620656 (+114001us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:20.705661 (+ 85005us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1104758}
I20260504 13:56:20.708343 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_4207c9e8019c4088b934376c5568d101"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:20.709860 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4207c9e8019c4088b934376c5568d101 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:20.706429 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd5f7d3f7c545d18fe19b64e4bc1de6" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:20.724457 9932 raft_consensus.cc:359] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.738870 9932 raft_consensus.cc:385] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:20.739256 9932 raft_consensus.cc:740] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:20.740046 9932 consensus_queue.cc:260] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.741110 9685 raft_consensus.cc:2468] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:20.755973 9539 raft_consensus.cc:2468] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:20.744992 9932 ts_tablet_manager.cc:1434] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.046s user 0.006s sys 0.000s
I20260504 13:56:20.757344 9932 tablet_bootstrap.cc:492] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:20.762384 9932 tablet_bootstrap.cc:654] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:20.758504 9473 leader_election.cc:304] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:20.766462 10318 raft_consensus.cc:2804] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:20.766844 10318 raft_consensus.cc:493] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:20.767381 10318 raft_consensus.cc:3060] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.771016 9338 catalog_manager.cc:5671] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:20.772221 9338 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet b8e4239942f34fb081a026a35740e465 (table default.Table_11d8ab61a6044154bb599875a0bc9004 [id=73e12524eb9e416782bf9f7fb58c93bd]). Expected version 1 got 0
I20260504 13:56:20.772377 10013 tablet_bootstrap.cc:492] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:20.772857 9338 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 48b193e958b4414aad29dd2bcee1a61b (table default.Table_2c39b14515bb46f6a0b0f1252d8e273d [id=e9e56499fd2243919308927b3614ae8e]). Expected version 1 got 0
I20260504 13:56:20.773561 10013 ts_tablet_manager.cc:1403] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.212s user 0.039s sys 0.027s
I20260504 13:56:20.773660 10318 raft_consensus.cc:515] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.777603 10013 raft_consensus.cc:359] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.778129 10013 raft_consensus.cc:385] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:20.778379 10013 raft_consensus.cc:740] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:20.780122 10369 consensus_queue.cc:1048] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:20.789788 10013 consensus_queue.cc:260] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.793774 9341 master_service.cc:954] ReplaceTablet: received request to replace tablet afad9329cf7745ba8168b2a93f998295 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:20.804023 10345 consensus_queue.cc:1048] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:20.807575 10318 leader_election.cc:290] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:20.778681 9663 tablet_service.cc:1511] Processing CreateTablet for tablet 975fa1a4975242a388c15fe73abea1c7 (DEFAULT_TABLE table=default.Table_a660204a2583425aa3ecd09c5cc01368 [id=4683a87561f14c049c1ed8eb68dc24d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.795995 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd5f7d3f7c545d18fe19b64e4bc1de6" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:20.815819 9684 raft_consensus.cc:3060] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:20.816197 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 975fa1a4975242a388c15fe73abea1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.823056 9684 raft_consensus.cc:2468] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:20.798010 9618 leader_election.cc:304] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:20.811862 9660 tablet_service.cc:1511] Processing CreateTablet for tablet 8702ce3ff1a845f3b70012855cd17efd (DEFAULT_TABLE table=default.Table_24dab2dd1f45402786d80cba80bbad82 [id=80ca58df22214450b894afea377e9c16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.831866 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8702ce3ff1a845f3b70012855cd17efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.816531 10013 ts_tablet_manager.cc:1434] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.043s user 0.005s sys 0.000s
I20260504 13:56:20.824144 9473 leader_election.cc:304] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:20.835630 10355 raft_consensus.cc:2804] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:20.836972 10355 raft_consensus.cc:697] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:20.837215 9513 tablet_service.cc:1511] Processing CreateTablet for tablet 975fa1a4975242a388c15fe73abea1c7 (DEFAULT_TABLE table=default.Table_a660204a2583425aa3ecd09c5cc01368 [id=4683a87561f14c049c1ed8eb68dc24d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.838081 9513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 975fa1a4975242a388c15fe73abea1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.837698 10355 consensus_queue.cc:237] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.842979 10013 tablet_bootstrap.cc:492] T b13303d2a14c4187b82da9f25ce9753e P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:20.843253 9474 consensus_peers.cc:597] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:20.844616 10369 consensus_queue.cc:1048] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:20.850498 10395 raft_consensus.cc:2804] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:20.850950 10395 raft_consensus.cc:697] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:20.851714 10395 consensus_queue.cc:237] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.853626 9515 tablet_service.cc:1511] Processing CreateTablet for tablet 8702ce3ff1a845f3b70012855cd17efd (DEFAULT_TABLE table=default.Table_24dab2dd1f45402786d80cba80bbad82 [id=80ca58df22214450b894afea377e9c16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.854506 9515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8702ce3ff1a845f3b70012855cd17efd. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:20.861107 9618 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 63177.6us
I20260504 13:56:20.859004 10013 tablet_bootstrap.cc:654] T b13303d2a14c4187b82da9f25ce9753e P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:20.872255 10369 consensus_queue.cc:1048] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:20.708401 9931 tablet_bootstrap.cc:492] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:20.891569 9931 ts_tablet_manager.cc:1403] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.483s user 0.009s sys 0.000s
I20260504 13:56:20.893920 9931 raft_consensus.cc:359] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.894501 9931 raft_consensus.cc:385] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:20.894735 9931 raft_consensus.cc:740] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:20.712112 9812 tablet_service.cc:1511] Processing CreateTablet for tablet 975fa1a4975242a388c15fe73abea1c7 (DEFAULT_TABLE table=default.Table_a660204a2583425aa3ecd09c5cc01368 [id=4683a87561f14c049c1ed8eb68dc24d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.895812 9812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 975fa1a4975242a388c15fe73abea1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.732645 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd5f7d3f7c545d18fe19b64e4bc1de6" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:20.793954 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd5f7d3f7c545d18fe19b64e4bc1de6" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:20.807582 9808 tablet_service.cc:1511] Processing CreateTablet for tablet 8702ce3ff1a845f3b70012855cd17efd (DEFAULT_TABLE table=default.Table_24dab2dd1f45402786d80cba80bbad82 [id=80ca58df22214450b894afea377e9c16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.880453 9810 tablet_service.cc:1511] Processing CreateTablet for tablet 7eaa40eb4b1c425a9edfb7eaba953b65 (DEFAULT_TABLE table=default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.901701 9810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eaa40eb4b1c425a9edfb7eaba953b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.904168 9808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8702ce3ff1a845f3b70012855cd17efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.905073 9931 consensus_queue.cc:260] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:20.909138 9931 ts_tablet_manager.cc:1434] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260504 13:56:20.911904 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet afee197774db411d97468913a207703a (table default.Table_91c2d3ed5c5c42758c87f1f726af88c0 [id=a2697dba5f38435c81dc52316712917d]) successfully deleted
I20260504 13:56:20.912889 10003 tablet_replica.cc:333] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:20.913722 10003 raft_consensus.cc:2243] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:20.915503 9931 tablet_bootstrap.cc:492] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:20.920557 9340 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_d6ee06783219496b94ecc1a88d13b0da" } modify_external_catalogs: true
I20260504 13:56:20.921038 9340 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_d6ee06783219496b94ecc1a88d13b0da" } modify_external_catalogs: true
I20260504 13:56:20.921150 9666 tablet_service.cc:1511] Processing CreateTablet for tablet 7eaa40eb4b1c425a9edfb7eaba953b65 (DEFAULT_TABLE table=default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.922089 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eaa40eb4b1c425a9edfb7eaba953b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.923969 10003 raft_consensus.cc:2272] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:20.923930 9814 tablet_service.cc:1511] Processing CreateTablet for tablet 3404ba04d3ec4a44a5e625536b941b58 (DEFAULT_TABLE table=default.Table_7db59fa280b6442b99cf24677b57df83 [id=fdb915ebe83443fd8d2f7f287385f6a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.925155 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3404ba04d3ec4a44a5e625536b941b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.926815 10013 tablet_bootstrap.cc:492] T b13303d2a14c4187b82da9f25ce9753e P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:20.932915 10013 ts_tablet_manager.cc:1403] T b13303d2a14c4187b82da9f25ce9753e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.090s user 0.010s sys 0.004s
I20260504 13:56:20.935120 10013 raft_consensus.cc:359] T b13303d2a14c4187b82da9f25ce9753e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.935866 10013 raft_consensus.cc:385] T b13303d2a14c4187b82da9f25ce9753e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:20.936106 10013 raft_consensus.cc:740] T b13303d2a14c4187b82da9f25ce9753e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:20.936647 10013 consensus_queue.cc:260] T b13303d2a14c4187b82da9f25ce9753e P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:20.937438 10003 ts_tablet_manager.cc:1916] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:20.939433 9335 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 70caac2e256d419dbc2e70cb5dfffe97
I20260504 13:56:20.940582 10013 ts_tablet_manager.cc:1434] T b13303d2a14c4187b82da9f25ce9753e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260504 13:56:20.940680 9335 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35768 (request call id 180) took 2199 ms. Trace:
I20260504 13:56:20.941051 9665 tablet_service.cc:1558] Processing DeleteTablet for tablet 70caac2e256d419dbc2e70cb5dfffe97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:18 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:20.941418 9813 tablet_service.cc:1558] Processing DeleteTablet for tablet 70caac2e256d419dbc2e70cb5dfffe97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:18 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:20.943701 10013 tablet_bootstrap.cc:492] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:20.940989 9335 rpcz_store.cc:276] 0504 13:56:18.740824 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.741005 (+ 181us) service_pool.cc:224] Handling call
0504 13:56:18.799941 (+ 58936us) spinlock_profiling.cc:243] Waited 58.1 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b0c46 00007fc27b2c2024 00007fc27b2c1fb2 00007fc27b2c1ee1 00007fc27b2c1e5c 00007fc27b2c1112 00007fc27958bbcc 00007fc27958b006
0504 13:56:18.801032 (+ 1091us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:18.801905 (+ 873us) catalog_manager.cc:2815] Modifying in-memory table state
0504 13:56:18.802059 (+ 154us) catalog_manager.cc:2824] Locking tablets
0504 13:56:20.188300 (+1386241us) catalog_manager.cc:2838] Removing table and tablets from system table
0504 13:56:20.541695 (+353395us) catalog_manager.cc:2856] Removing table from by-name map
0504 13:56:20.541891 (+ 196us) catalog_manager.cc:2872] Committing in-memory state
0504 13:56:20.939344 (+397453us) catalog_manager.cc:2876] Aborting table tasks
0504 13:56:20.940630 (+ 1286us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:20.502421 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:20.502933 (+ 512us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:20.502967 (+ 34us) write_op.cc:435] Acquired schema lock
0504 13:56:20.502986 (+ 19us) tablet.cc:662] Decoding operations
0504 13:56:20.503372 (+ 386us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:20.503409 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:20.503428 (+ 19us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:20.503654 (+ 226us) tablet.cc:701] Row locks acquired
0504 13:56:20.503667 (+ 13us) write_op.cc:260] PREPARE: finished
0504 13:56:20.503764 (+ 97us) write_op.cc:270] Start()
0504 13:56:20.503837 (+ 73us) write_op.cc:276] Timestamp: P: 1777902980503710 usec, L: 0
0504 13:56:20.503852 (+ 15us) op_driver.cc:348] REPLICATION: starting
0504 13:56:20.504295 (+ 443us) log.cc:844] Serialized 762 byte log entry
0504 13:56:20.539388 (+ 35093us) op_driver.cc:464] REPLICATION: finished
0504 13:56:20.539776 (+ 388us) write_op.cc:301] APPLY: starting
0504 13:56:20.539828 (+ 52us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:20.539938 (+ 110us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:20.540041 (+ 103us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:20.540303 (+ 262us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:20.540456 (+ 153us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:20.540497 (+ 41us) write_op.cc:312] APPLY: finished
0504 13:56:20.540666 (+ 169us) log.cc:844] Serialized 29 byte log entry
0504 13:56:20.540810 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:20.540886 (+ 76us) write_op.cc:454] Released schema lock
0504 13:56:20.540921 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:20.552333 (+ 11412us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:20.552356 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":133634816,"child_traces":[["op",{"apply.queue_time_us":244,"apply.run_cpu_time_us":1335,"apply.run_wall_time_us":1335,"num_ops":2,"prepare.queue_time_us":312468,"prepare.run_cpu_time_us":2236,"prepare.run_wall_time_us":49917,"raft.queue_time_us":15696,"raft.run_cpu_time_us":618,"raft.run_wall_time_us":617,"replication_time_us":35431}]]}
I20260504 13:56:20.945963 9336 master_service.cc:954] ReplaceTablet: received request to replace tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:20.948876 9665 tablet_service.cc:1511] Processing CreateTablet for tablet 3404ba04d3ec4a44a5e625536b941b58 (DEFAULT_TABLE table=default.Table_7db59fa280b6442b99cf24677b57df83 [id=fdb915ebe83443fd8d2f7f287385f6a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.949749 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3404ba04d3ec4a44a5e625536b941b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.949944 9813 tablet_service.cc:1511] Processing CreateTablet for tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1 (DEFAULT_TABLE table=default.Table_7f9a1296bc424f7aa21625c670da0dd0 [id=a62d0dfdbe7741248a4d5d6614c712eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.950831 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:20.953954 9343 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:38691 (request call id 36) took 2082 ms (client timeout 1000 ms). Trace:
W20260504 13:56:20.954326 9343 rpcz_store.cc:269] 0504 13:56:18.872027 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.895245 (+ 23218us) service_pool.cc:224] Handling call
0504 13:56:20.953895 (+2058650us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":6}
I20260504 13:56:20.963917 9932 tablet_bootstrap.cc:492] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:20.964394 9932 ts_tablet_manager.cc:1403] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.207s user 0.002s sys 0.008s
I20260504 13:56:20.965669 9944 ts_tablet_manager.cc:1929] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:20.966006 9944 log.cc:1199] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/b3793fe8f1ec43ad809403c4593f5ef6
I20260504 13:56:20.966794 9944 ts_tablet_manager.cc:1950] T b3793fe8f1ec43ad809403c4593f5ef6 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:20.973433 9931 tablet_bootstrap.cc:654] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:20.974440 10013 tablet_bootstrap.cc:654] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:20.975663 9664 tablet_service.cc:1511] Processing CreateTablet for tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1 (DEFAULT_TABLE table=default.Table_7f9a1296bc424f7aa21625c670da0dd0 [id=a62d0dfdbe7741248a4d5d6614c712eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.979874 9662 tablet_service.cc:1511] Processing CreateTablet for tablet 7c9030802900499a90c4dbaf5c6936a4 (DEFAULT_TABLE table=default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.980358 9343 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_bdb22fcd488647e0bfed4170cb70365d" } new_table_name: "default.Table_fd3cceda3257414892a1f1e69d4e6344" modify_external_catalogs: true
I20260504 13:56:20.981781 9664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:20.981812 9335 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_7f912e1d4381428e9572cbb4b13f4eec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:20.983743 9662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c9030802900499a90c4dbaf5c6936a4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:20.985623 9335 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f912e1d4381428e9572cbb4b13f4eec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:20.990464 9661 tablet_service.cc:1511] Processing CreateTablet for tablet e5d63f665166466dba1e7e36c5e19506 (DEFAULT_TABLE table=default.Table_363d6634e42e4dd6a5686c1f0e401ef2 [id=d0d8b54fb69540589420b87f90b4c823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:20.991531 9661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d63f665166466dba1e7e36c5e19506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.006451 9659 tablet_service.cc:1511] Processing CreateTablet for tablet 9077fb9bb1544a2cbeb96f0d11c400db (DEFAULT_TABLE table=default.Table_06ecf48a04d94ac8a7e954ff2c260dbf [id=e11be55403d24c8bb35f4708da2e9a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.010123 10003 ts_tablet_manager.cc:1929] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:21.013823 9658 tablet_service.cc:1511] Processing CreateTablet for tablet 274bb204041c4e1085395c7bd938aaf1 (DEFAULT_TABLE table=default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.016542 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 101) took 2560 ms. Trace:
I20260504 13:56:21.016840 9944 rpcz_store.cc:276] 0504 13:56:18.456081 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.458220 (+ 2139us) service_pool.cc:224] Handling call
0504 13:56:20.148623 (+1690403us) ts_tablet_manager.cc:1151] Deleting tablet b3793fe8f1ec43ad809403c4593f5ef6
0504 13:56:20.148661 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:20.453586 (+304925us) spinlock_profiling.cc:243] Waited 36.9 ms on lock 0x7b58000a01d0. stack: 00007f6906f51c83 00007f6906f51b80 00007f6906ac2574 000000000050e094 000000000050e029 00007f690dfa9951 00007f690df9c48a 00007f690e01e827 00007f690e048765 00007f690e46b7bc 00007f690f0c3cca 00007f690f0f4990 00007f690f0daefd 00007f690f0daea9 00007f690f0dae39 00007f690f0dae01
0504 13:56:20.799607 (+346021us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:20.965565 (+165958us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.016486 (+ 50921us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57542,"spinlock_wait_cycles":84974336,"tablet-delete.queue_time_us":1689924}
I20260504 13:56:21.019901 10003 log.cc:1199] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/cdb86f6370bf4c5f9c488d85efb67ef7
I20260504 13:56:21.020645 10003 ts_tablet_manager.cc:1950] T cdb86f6370bf4c5f9c488d85efb67ef7 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:21.023222 9657 tablet_service.cc:1511] Processing CreateTablet for tablet 8a985338790e4887af38c49ba013417f (DEFAULT_TABLE table=default.Table_5636294717ff4c63939c2cf9facfc788 [id=6acf451855024161b7826094f8387a9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.024065 10003 tablet_replica.cc:333] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:21.024312 9657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a985338790e4887af38c49ba013417f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.024734 10003 raft_consensus.cc:2243] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:21.025120 10003 raft_consensus.cc:2272] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:21.029136 9659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9077fb9bb1544a2cbeb96f0d11c400db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.032398 9658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274bb204041c4e1085395c7bd938aaf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.042539 9656 tablet_service.cc:1511] Processing CreateTablet for tablet 60d3cc59ffe947fbb57b7d3301ccbac1 (DEFAULT_TABLE table=default.Table_28052c777ee944b79b802d28b415c24e [id=994b65b95f66440283f852cd9dd9ae0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.044132 9656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60d3cc59ffe947fbb57b7d3301ccbac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.049710 9944 tablet_replica.cc:333] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:21.050457 9944 raft_consensus.cc:2243] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:21.050894 9944 raft_consensus.cc:2272] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:21.052663 9944 ts_tablet_manager.cc:1916] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:21.055925 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet b3793fe8f1ec43ad809403c4593f5ef6 (table default.Table_61add6c7b0dd46b08c0e39ad6d697e8c [id=1749c1a022544b1185fb3d4a4a530afd]) successfully deleted
I20260504 13:56:21.057369 10003 ts_tablet_manager.cc:1916] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:21.058636 10395 raft_consensus.cc:493] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:21.059105 10395 raft_consensus.cc:515] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.060935 10395 leader_election.cc:290] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:21.069478 9944 ts_tablet_manager.cc:1929] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:21.069797 9944 log.cc:1199] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/d7e556f81ec940f1aa3c55b3a25f31ab
I20260504 13:56:21.070890 9659 tablet_service.cc:1511] Processing CreateTablet for tablet 184fefa651fc411fb5107cc923c02535 (DEFAULT_TABLE table=default.Table_f8a23a8252f84cacb13b338f9d9dc9fc [id=39be2c4d16aa41c0832bfeb23269243a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.071386 9932 raft_consensus.cc:359] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.072024 9932 raft_consensus.cc:385] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:21.072079 9659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 184fefa651fc411fb5107cc923c02535. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.072316 9932 raft_consensus.cc:740] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:21.072886 9932 consensus_queue.cc:260] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.075637 9932 ts_tablet_manager.cc:1434] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.111s user 0.006s sys 0.000s
I20260504 13:56:21.076407 9932 tablet_bootstrap.cc:492] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:21.084333 10013 tablet_bootstrap.cc:492] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:21.084842 10013 ts_tablet_manager.cc:1403] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.144s user 0.005s sys 0.005s
I20260504 13:56:21.089131 9341 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet afad9329cf7745ba8168b2a93f998295
I20260504 13:56:21.090278 9341 catalog_manager.cc:6741] ReplaceTablet: tablet afad9329cf7745ba8168b2a93f998295 deleted and replaced by tablet af5c5a6cae4e4b6ea3eece2e21a587ec
I20260504 13:56:21.091964 9660 tablet_service.cc:1558] Processing DeleteTablet for tablet afad9329cf7745ba8168b2a93f998295 with delete_type TABLET_DATA_DELETED (replaced by tablet af5c5a6cae4e4b6ea3eece2e21a587ec) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:21.105419 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_1e6b4c9e684e45b3b2510d3979c1b5c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:21.107278 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e6b4c9e684e45b3b2510d3979c1b5c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:21.113664 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet cdb86f6370bf4c5f9c488d85efb67ef7 (table default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]) successfully deleted
I20260504 13:56:21.114732 9811 tablet_service.cc:1511] Processing CreateTablet for tablet 9077fb9bb1544a2cbeb96f0d11c400db (DEFAULT_TABLE table=default.Table_06ecf48a04d94ac8a7e954ff2c260dbf [id=e11be55403d24c8bb35f4708da2e9a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.116044 9807 tablet_service.cc:1511] Processing CreateTablet for tablet 274bb204041c4e1085395c7bd938aaf1 (DEFAULT_TABLE table=default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.116312 9811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9077fb9bb1544a2cbeb96f0d11c400db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.117043 9807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274bb204041c4e1085395c7bd938aaf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.125566 9340 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 2951cbd2df4f402eabeef323d121ce1b
I20260504 13:56:21.126741 9805 tablet_service.cc:1511] Processing CreateTablet for tablet 60d3cc59ffe947fbb57b7d3301ccbac1 (DEFAULT_TABLE table=default.Table_28052c777ee944b79b802d28b415c24e [id=994b65b95f66440283f852cd9dd9ae0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.127753 9805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60d3cc59ffe947fbb57b7d3301ccbac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.127843 9660 tablet_service.cc:1558] Processing DeleteTablet for tablet 2951cbd2df4f402eabeef323d121ce1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:20 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:21.132429 9804 tablet_service.cc:1511] Processing CreateTablet for tablet 184fefa651fc411fb5107cc923c02535 (DEFAULT_TABLE table=default.Table_f8a23a8252f84cacb13b338f9d9dc9fc [id=39be2c4d16aa41c0832bfeb23269243a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.133811 9813 tablet_service.cc:1558] Processing DeleteTablet for tablet 2951cbd2df4f402eabeef323d121ce1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:20 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:21.134663 9804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 184fefa651fc411fb5107cc923c02535. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.136930 9803 tablet_service.cc:1558] Processing DeleteTablet for tablet afad9329cf7745ba8168b2a93f998295 with delete_type TABLET_DATA_DELETED (replaced by tablet af5c5a6cae4e4b6ea3eece2e21a587ec) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:21.139873 9806 tablet_service.cc:1511] Processing CreateTablet for tablet 8a985338790e4887af38c49ba013417f (DEFAULT_TABLE table=default.Table_5636294717ff4c63939c2cf9facfc788 [id=6acf451855024161b7826094f8387a9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.141729 9806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a985338790e4887af38c49ba013417f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.144295 9809 tablet_service.cc:1511] Processing CreateTablet for tablet 7c9030802900499a90c4dbaf5c6936a4 (DEFAULT_TABLE table=default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.145402 9809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c9030802900499a90c4dbaf5c6936a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.151566 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b13303d2a14c4187b82da9f25ce9753e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:21.153393 9620 leader_election.cc:343] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:21.165730 10395 raft_consensus.cc:493] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:21.166177 10395 raft_consensus.cc:515] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.167950 10395 leader_election.cc:290] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:21.170131 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50e129980cd84f6ca22cbf2addb06230" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:21.171780 9620 leader_election.cc:343] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:21.176146 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_3e6fece0bfdb447dae80b4598b049417"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:21.177906 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e6fece0bfdb447dae80b4598b049417 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:21.070420 9944 ts_tablet_manager.cc:1950] T d7e556f81ec940f1aa3c55b3a25f31ab P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:21.194700 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 102) took 2735 ms. Trace:
I20260504 13:56:21.218086 10003 ts_tablet_manager.cc:1929] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:21.218447 10003 log.cc:1199] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/2b50573310a04f3694eeab2fef0fb540
I20260504 13:56:21.219034 10003 ts_tablet_manager.cc:1950] T 2b50573310a04f3694eeab2fef0fb540 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:21.221236 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 2b50573310a04f3694eeab2fef0fb540 (table default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]) successfully deleted
W20260504 13:56:21.225723 9330 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet afee197774db411d97468913a207703a (table default.Table_91c2d3ed5c5c42758c87f1f726af88c0 [id=a2697dba5f38435c81dc52316712917d], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet afee197774db411d97468913a207703a consensus state has no leader
I20260504 13:56:21.209800 10395 raft_consensus.cc:493] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:21.235276 9944 rpcz_store.cc:276] 0504 13:56:18.458974 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.485163 (+ 26189us) service_pool.cc:224] Handling call
0504 13:56:21.049578 (+2564415us) ts_tablet_manager.cc:1151] Deleting tablet d7e556f81ec940f1aa3c55b3a25f31ab
0504 13:56:21.049615 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:21.060158 (+ 10543us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.069386 (+ 9228us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.194641 (+125255us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2558354}
I20260504 13:56:21.235442 10395 raft_consensus.cc:515] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.236539 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet d7e556f81ec940f1aa3c55b3a25f31ab (table default.Table_ea18464634544273a0bbb10fce8b88ba [id=2388ca57302443e48f66ac3a8db39000]) successfully deleted
I20260504 13:56:21.237962 10395 leader_election.cc:290] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:21.234920 9932 tablet_bootstrap.cc:654] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:21.255609 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4c185332a24d96bc3d0fddc0d20693" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:21.256835 9620 leader_election.cc:343] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:21.261118 9814 tablet_service.cc:1511] Processing CreateTablet for tablet e5d63f665166466dba1e7e36c5e19506 (DEFAULT_TABLE table=default.Table_363d6634e42e4dd6a5686c1f0e401ef2 [id=d0d8b54fb69540589420b87f90b4c823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.262511 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d63f665166466dba1e7e36c5e19506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.268540 10382 raft_consensus.cc:493] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:21.268985 10382 raft_consensus.cc:515] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:21.270793 10382 leader_election.cc:290] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:21.271915 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70caac2e256d419dbc2e70cb5dfffe97" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:21.272720 9684 raft_consensus.cc:2393] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
W20260504 13:56:21.276350 9473 consensus_peers.cc:597] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c43ff1d759a40d68344f25f066416ac. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:21.281507 9503 tablet_service.cc:1558] Processing DeleteTablet for tablet 2951cbd2df4f402eabeef323d121ce1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:20 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:21.282725 9944 tablet_replica.cc:333] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:21.282330 9507 tablet_service.cc:1511] Processing CreateTablet for tablet 8a985338790e4887af38c49ba013417f (DEFAULT_TABLE table=default.Table_5636294717ff4c63939c2cf9facfc788 [id=6acf451855024161b7826094f8387a9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.282343 9508 tablet_service.cc:1511] Processing CreateTablet for tablet 274bb204041c4e1085395c7bd938aaf1 (DEFAULT_TABLE table=default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]), partition=RANGE (key) PARTITION UNBOUNDED
W20260504 13:56:21.284093 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 407249us
I20260504 13:56:21.284291 9505 tablet_service.cc:1511] Processing CreateTablet for tablet 184fefa651fc411fb5107cc923c02535 (DEFAULT_TABLE table=default.Table_f8a23a8252f84cacb13b338f9d9dc9fc [id=39be2c4d16aa41c0832bfeb23269243a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.285583 9932 tablet_bootstrap.cc:492] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:21.286019 9932 ts_tablet_manager.cc:1403] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.210s user 0.010s sys 0.000s
I20260504 13:56:21.286094 10395 raft_consensus.cc:493] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:21.286600 10395 raft_consensus.cc:515] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:21.288424 10395 leader_election.cc:290] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
W20260504 13:56:21.289088 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 50e129980cd84f6ca22cbf2addb06230 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50e129980cd84f6ca22cbf2addb06230 already in progress: creating tablet
I20260504 13:56:21.289381 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030b35c3409143108c34ab1f0ee1b96d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:21.282943 9504 tablet_service.cc:1558] Processing DeleteTablet for tablet afad9329cf7745ba8168b2a93f998295 with delete_type TABLET_DATA_DELETED (replaced by tablet af5c5a6cae4e4b6ea3eece2e21a587ec) from {username='slave'} at 127.0.0.1:52642
W20260504 13:56:21.290768 9620 leader_election.cc:343] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:21.291513 9508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274bb204041c4e1085395c7bd938aaf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.292797 9932 raft_consensus.cc:359] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:21.293610 9932 raft_consensus.cc:385] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:21.293910 9932 raft_consensus.cc:740] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:21.296833 10003 tablet_replica.cc:333] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:21.297111 9513 tablet_service.cc:1558] Processing DeleteTablet for tablet 70caac2e256d419dbc2e70cb5dfffe97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:18 UTC) from {username='slave'} at 127.0.0.1:52642
W20260504 13:56:21.298236 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet c34630d55f4a4e19b33ddf4c13995a5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c34630d55f4a4e19b33ddf4c13995a5b already in progress: creating tablet
I20260504 13:56:21.300768 9512 tablet_service.cc:1511] Processing CreateTablet for tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1 (DEFAULT_TABLE table=default.Table_7f9a1296bc424f7aa21625c670da0dd0 [id=a62d0dfdbe7741248a4d5d6614c712eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.302431 9932 consensus_queue.cc:260] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:21.303579 9509 tablet_service.cc:1511] Processing CreateTablet for tablet 9077fb9bb1544a2cbeb96f0d11c400db (DEFAULT_TABLE table=default.Table_06ecf48a04d94ac8a7e954ff2c260dbf [id=e11be55403d24c8bb35f4708da2e9a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.316592 9507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a985338790e4887af38c49ba013417f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.318008 9506 tablet_service.cc:1511] Processing CreateTablet for tablet 60d3cc59ffe947fbb57b7d3301ccbac1 (DEFAULT_TABLE table=default.Table_28052c777ee944b79b802d28b415c24e [id=994b65b95f66440283f852cd9dd9ae0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.322751 9511 tablet_service.cc:1511] Processing CreateTablet for tablet 7c9030802900499a90c4dbaf5c6936a4 (DEFAULT_TABLE table=default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.319970 9510 tablet_service.cc:1511] Processing CreateTablet for tablet e5d63f665166466dba1e7e36c5e19506 (DEFAULT_TABLE table=default.Table_363d6634e42e4dd6a5686c1f0e401ef2 [id=d0d8b54fb69540589420b87f90b4c823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.324802 9932 ts_tablet_manager.cc:1434] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20260504 13:56:21.333276 10003 raft_consensus.cc:2243] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:21.333877 10003 raft_consensus.cc:2272] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:21.339198 9505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 184fefa651fc411fb5107cc923c02535. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.340569 9506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60d3cc59ffe947fbb57b7d3301ccbac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.347457 9509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9077fb9bb1544a2cbeb96f0d11c400db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.348738 9510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d63f665166466dba1e7e36c5e19506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.353768 9932 tablet_bootstrap.cc:492] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:21.355391 9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.356861 9511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c9030802900499a90c4dbaf5c6936a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.360664 9931 tablet_bootstrap.cc:492] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:21.361105 9931 ts_tablet_manager.cc:1403] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.446s user 0.006s sys 0.022s
I20260504 13:56:21.367133 9931 raft_consensus.cc:359] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:21.367547 10003 ts_tablet_manager.cc:1916] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:21.367767 9931 raft_consensus.cc:385] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:21.368053 9931 raft_consensus.cc:740] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:21.368942 9931 consensus_queue.cc:260] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:21.373406 9931 ts_tablet_manager.cc:1434] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260504 13:56:21.376482 9931 tablet_bootstrap.cc:492] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:21.368301 9474 consensus_peers.cc:597] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:21.378113 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 50298.8us
I20260504 13:56:21.381870 9931 tablet_bootstrap.cc:654] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:21.382316 9474 leader_election.cc:343] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:21.383415 9474 leader_election.cc:343] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:21.394454 9931 tablet_bootstrap.cc:492] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:21.394806 9931 ts_tablet_manager.cc:1403] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20260504 13:56:21.397980 9931 raft_consensus.cc:359] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:21.398555 9931 raft_consensus.cc:385] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:21.398798 9931 raft_consensus.cc:740] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:21.399336 9931 consensus_queue.cc:260] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:21.401010 9931 ts_tablet_manager.cc:1434] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260504 13:56:21.401741 9931 tablet_bootstrap.cc:492] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:21.283453 9944 raft_consensus.cc:2243] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:21.406324 9944 raft_consensus.cc:2272] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:21.408089 9944 ts_tablet_manager.cc:1916] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:21.411808 9931 tablet_bootstrap.cc:654] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:21.419801 9519 tablet_service.cc:1558] Processing DeleteTablet for tablet a43cdb767fd14ea09b4db4d2881b69aa with delete_type TABLET_DATA_DELETED (replaced by tablet 7c9030802900499a90c4dbaf5c6936a4) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:21.419854 9517 tablet_service.cc:1558] Processing DeleteTablet for tablet 65052a5c28c147a385144317e21eea41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:21.425510 9932 tablet_bootstrap.cc:654] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:21.435863 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030b35c3409143108c34ab1f0ee1b96d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:21.436852 9536 raft_consensus.cc:2468] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:21.438340 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b13303d2a14c4187b82da9f25ce9753e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:21.438910 9539 raft_consensus.cc:2468] T b13303d2a14c4187b82da9f25ce9753e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:21.439977 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4c185332a24d96bc3d0fddc0d20693" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:21.456703 9518 tablet_service.cc:1511] Processing CreateTablet for tablet 7eaa40eb4b1c425a9edfb7eaba953b65 (DEFAULT_TABLE table=default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.458305 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eaa40eb4b1c425a9edfb7eaba953b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.459758 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50e129980cd84f6ca22cbf2addb06230" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:21.464795 10013 raft_consensus.cc:359] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.465483 10013 raft_consensus.cc:385] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:21.465718 10013 raft_consensus.cc:740] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:21.466223 10013 consensus_queue.cc:260] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:21.470402 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 86148.1us
I20260504 13:56:21.473640 10355 raft_consensus.cc:493] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:21.474223 10355 raft_consensus.cc:515] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.476320 10355 leader_election.cc:290] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:21.478994 10003 ts_tablet_manager.cc:1929] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:21.479453 10003 log.cc:1199] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/70caac2e256d419dbc2e70cb5dfffe97
I20260504 13:56:21.480206 10003 ts_tablet_manager.cc:1950] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:21.482045 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 70caac2e256d419dbc2e70cb5dfffe97 (table default.Table_c5584202645f4ef78e4f6e00aa2bf2c1 [id=2665806fb81f4eda8981fede847cbfcf]) successfully deleted
I20260504 13:56:21.489120 9520 tablet_service.cc:1511] Processing CreateTablet for tablet 3404ba04d3ec4a44a5e625536b941b58 (DEFAULT_TABLE table=default.Table_7db59fa280b6442b99cf24677b57df83 [id=fdb915ebe83443fd8d2f7f287385f6a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:21.492553 10003 tablet_replica.cc:333] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:21.493424 10003 raft_consensus.cc:2243] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:21.494378 10003 raft_consensus.cc:2272] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:21.503093 9944 ts_tablet_manager.cc:1929] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:21.503484 9944 log.cc:1199] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/eb7f58645c76441589368072274ebc8f
I20260504 13:56:21.504172 9944 ts_tablet_manager.cc:1950] T eb7f58645c76441589368072274ebc8f P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:21.507368 10003 ts_tablet_manager.cc:1916] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:21.516016 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 108) took 2943 ms. Trace:
I20260504 13:56:21.516378 9944 rpcz_store.cc:276] 0504 13:56:18.572230 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.572342 (+ 112us) service_pool.cc:224] Handling call
0504 13:56:21.282564 (+2710222us) ts_tablet_manager.cc:1151] Deleting tablet eb7f58645c76441589368072274ebc8f
0504 13:56:21.282612 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:21.459985 (+177373us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.502974 (+ 42989us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.515949 (+ 12975us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2709699}
I20260504 13:56:21.519217 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet eb7f58645c76441589368072274ebc8f (table default.Table_d5cc86d81fb043319576af0a39c643a8 [id=7af4adcf5ee44bad89afa2e4ded416fc]) successfully deleted
I20260504 13:56:21.507768 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3404ba04d3ec4a44a5e625536b941b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:21.504071 10013 ts_tablet_manager.cc:1434] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.419s user 0.006s sys 0.001s
I20260504 13:56:21.528431 10013 tablet_bootstrap.cc:492] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:21.528703 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 111) took 2841 ms. Trace:
I20260504 13:56:21.528993 9944 rpcz_store.cc:276] 0504 13:56:18.687137 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.756077 (+ 68940us) service_pool.cc:224] Handling call
0504 13:56:21.528441 (+2772364us) ts_tablet_manager.cc:1151] Deleting tablet 39f080a0296b44988e8e51e68453a551
0504 13:56:21.528481 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:21.528652 (+ 171us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2771834}
W20260504 13:56:21.530059 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 39f080a0296b44988e8e51e68453a551 because the tablet was not found. No further retry: Not found: Tablet not found: 39f080a0296b44988e8e51e68453a551
I20260504 13:56:21.538511 9944 tablet_replica.cc:333] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:21.539326 9944 raft_consensus.cc:2243] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:21.539745 9944 pending_rounds.cc:70] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6: Trying to abort 1 pending ops.
I20260504 13:56:21.539963 9944 pending_rounds.cc:77] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290604837711872 op_type: NO_OP noop_request { }
I20260504 13:56:21.540277 9944 raft_consensus.cc:2889] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:21.540582 9944 raft_consensus.cc:2272] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:21.496806 10391 ts_tablet_manager.cc:1929] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:21.547610 10391 log.cc:1199] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/2b50573310a04f3694eeab2fef0fb540
I20260504 13:56:21.549217 9944 ts_tablet_manager.cc:1916] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:21.566038 9944 ts_tablet_manager.cc:1929] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:21.566361 9944 log.cc:1199] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/65052a5c28c147a385144317e21eea41
I20260504 13:56:21.567395 9834 raft_consensus.cc:3060] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:21.567028 9944 ts_tablet_manager.cc:1950] T 65052a5c28c147a385144317e21eea41 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:21.576829 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 112) took 2820 ms. Trace:
I20260504 13:56:21.577176 9944 rpcz_store.cc:276] 0504 13:56:18.756909 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.757007 (+ 98us) service_pool.cc:224] Handling call
0504 13:56:21.538365 (+2781358us) ts_tablet_manager.cc:1151] Deleting tablet 65052a5c28c147a385144317e21eea41
0504 13:56:21.538407 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:21.559972 (+ 21565us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.565941 (+ 5969us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.576778 (+ 10837us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2780875}
I20260504 13:56:21.578439 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 65052a5c28c147a385144317e21eea41 (table default.Table_038807318bcc4110b77de9582f6c42f5 [id=c19c0f3f26934ad6bbf2d4202cdaa542]) successfully deleted
I20260504 13:56:21.581419 9944 tablet_replica.cc:333] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:21.582156 9944 raft_consensus.cc:2243] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:21.582669 9944 raft_consensus.cc:2272] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:21.583348 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34630d55f4a4e19b33ddf4c13995a5b" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:21.586747 9944 ts_tablet_manager.cc:1916] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:21.588878 10391 ts_tablet_manager.cc:1950] T 2b50573310a04f3694eeab2fef0fb540 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:21.620080 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 124) took 1282 ms. Trace:
I20260504 13:56:21.620468 9944 ts_tablet_manager.cc:1929] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:21.620798 9944 log.cc:1199] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/cdb86f6370bf4c5f9c488d85efb67ef7
I20260504 13:56:21.621399 9944 ts_tablet_manager.cc:1950] T cdb86f6370bf4c5f9c488d85efb67ef7 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:21.620529 10391 rpcz_store.cc:276] 0504 13:56:20.337973 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:20.338095 (+ 122us) service_pool.cc:224] Handling call
0504 13:56:20.405095 (+ 67000us) ts_tablet_manager.cc:1151] Deleting tablet 2b50573310a04f3694eeab2fef0fb540
0504 13:56:20.405141 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:20.829817 (+424676us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.496586 (+666769us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.620019 (+123433us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":66468,"thread_start_us":68780,"threads_started":1}
I20260504 13:56:21.622519 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 126) took 1177 ms. Trace:
I20260504 13:56:21.624284 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 2b50573310a04f3694eeab2fef0fb540 (table default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]) successfully deleted
I20260504 13:56:21.623152 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 113) took 2865 ms. Trace:
I20260504 13:56:21.629026 9944 rpcz_store.cc:276] 0504 13:56:18.757726 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.757805 (+ 79us) service_pool.cc:224] Handling call
0504 13:56:21.581282 (+2823477us) ts_tablet_manager.cc:1151] Deleting tablet cdb86f6370bf4c5f9c488d85efb67ef7
0504 13:56:21.581319 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:21.609047 (+ 27728us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.620360 (+ 11313us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.623091 (+ 2731us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2823043}
I20260504 13:56:21.630085 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet cdb86f6370bf4c5f9c488d85efb67ef7 (table default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]) successfully deleted
I20260504 13:56:21.631561 9944 tablet_replica.cc:333] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:21.632292 9944 raft_consensus.cc:2243] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:21.634171 9944 raft_consensus.cc:2272] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:21.637079 9944 ts_tablet_manager.cc:1916] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:21.633884 10003 ts_tablet_manager.cc:1929] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260504 13:56:21.643854 10003 log.cc:1199] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/2951cbd2df4f402eabeef323d121ce1b
I20260504 13:56:21.645537 10003 ts_tablet_manager.cc:1950] T 2951cbd2df4f402eabeef323d121ce1b P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:21.649121 9336 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 1cf2b59d13da40a2b2160ebdc0b2eb80
I20260504 13:56:21.650506 9336 catalog_manager.cc:6741] ReplaceTablet: tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 deleted and replaced by tablet a862a43c5ac443d9bac98e0472ca23b2
I20260504 13:56:21.651649 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 2951cbd2df4f402eabeef323d121ce1b (table default.Table_d6ee06783219496b94ecc1a88d13b0da [id=46ac1f0baae0492984e87927e4905343]) successfully deleted
I20260504 13:56:21.653964 9814 tablet_service.cc:1558] Processing DeleteTablet for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 with delete_type TABLET_DATA_DELETED (replaced by tablet a862a43c5ac443d9bac98e0472ca23b2) from {username='slave'} at 127.0.0.1:56380
W20260504 13:56:21.655891 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet afad9329cf7745ba8168b2a93f998295 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afad9329cf7745ba8168b2a93f998295 already in progress: creating tablet
I20260504 13:56:21.657976 9342 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 48b193e958b4414aad29dd2bcee1a61b (table default.Table_2c39b14515bb46f6a0b0f1252d8e273d [id=e9e56499fd2243919308927b3614ae8e]). Expected version 1 got 0
I20260504 13:56:21.658737 9342 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]). Expected version 1 got 0
I20260504 13:56:21.661204 10003 tablet_replica.cc:333] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:21.663304 10003 raft_consensus.cc:2243] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:21.665143 9659 tablet_service.cc:1558] Processing DeleteTablet for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 with delete_type TABLET_DATA_DELETED (replaced by tablet a862a43c5ac443d9bac98e0472ca23b2) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:21.664439 10003 raft_consensus.cc:2272] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:21.670449 9620 consensus_peers.cc:597] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:21.672921 9932 tablet_bootstrap.cc:492] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:21.673404 9932 ts_tablet_manager.cc:1403] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.320s user 0.027s sys 0.020s
I20260504 13:56:21.675314 9931 tablet_bootstrap.cc:492] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:21.675691 9931 ts_tablet_manager.cc:1403] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.274s user 0.027s sys 0.097s
I20260504 13:56:21.675384 9339 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet b8e4239942f34fb081a026a35740e465 (table default.Table_11d8ab61a6044154bb599875a0bc9004 [id=73e12524eb9e416782bf9f7fb58c93bd]). Expected version 1 got 0
I20260504 13:56:21.676039 9932 raft_consensus.cc:359] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.676666 9932 raft_consensus.cc:385] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:21.676914 9932 raft_consensus.cc:740] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:21.677632 9339 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]). Expected version 1 got 0
I20260504 13:56:21.677560 9932 consensus_queue.cc:260] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.677990 9931 raft_consensus.cc:359] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:21.678587 9931 raft_consensus.cc:385] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:21.678565 9339 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 48b193e958b4414aad29dd2bcee1a61b (table default.Table_2c39b14515bb46f6a0b0f1252d8e273d [id=e9e56499fd2243919308927b3614ae8e]). Expected version 1 got 0
I20260504 13:56:21.678838 9931 raft_consensus.cc:740] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:21.679617 10003 ts_tablet_manager.cc:1916] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:21.679419 9931 consensus_queue.cc:260] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:21.679636 9339 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:21.680310 9339 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 0be795f04dd54cf19869c937b91d0ddf (table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]). Expected version 1 got 0
I20260504 13:56:21.681707 9339 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet b13303d2a14c4187b82da9f25ce9753e (table default.Table_b32f665dddb043769e07076397f84a73 [id=f227576f105c4b1fbd459b3638ce684b]). Expected version 1 got 0
I20260504 13:56:21.690971 9944 ts_tablet_manager.cc:1929] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:21.691120 9931 ts_tablet_manager.cc:1434] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260504 13:56:21.691351 9944 log.cc:1199] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/07a428049fdb4a68aae729400e2d9674
I20260504 13:56:21.691917 9931 tablet_bootstrap.cc:492] T ab4c185332a24d96bc3d0fddc0d20693 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:21.691959 9944 ts_tablet_manager.cc:1950] T 07a428049fdb4a68aae729400e2d9674 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:21.693651 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 114) took 2316 ms. Trace:
I20260504 13:56:21.693913 9944 rpcz_store.cc:276] 0504 13:56:19.377483 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:19.380032 (+ 2549us) service_pool.cc:224] Handling call
0504 13:56:21.631417 (+2251385us) ts_tablet_manager.cc:1151] Deleting tablet 07a428049fdb4a68aae729400e2d9674
0504 13:56:21.631458 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:21.650585 (+ 19127us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.690864 (+ 40279us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.693602 (+ 2738us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2250879}
I20260504 13:56:21.694880 9944 tablet_replica.cc:333] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:21.706925 9931 tablet_bootstrap.cc:654] T ab4c185332a24d96bc3d0fddc0d20693 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:21.710817 9932 ts_tablet_manager.cc:1434] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.037s user 0.006s sys 0.001s
I20260504 13:56:21.716612 9931 tablet_bootstrap.cc:492] T ab4c185332a24d96bc3d0fddc0d20693 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:21.716981 9931 ts_tablet_manager.cc:1403] T ab4c185332a24d96bc3d0fddc0d20693 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.000s
I20260504 13:56:21.626246 10358 raft_consensus.cc:493] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:21.629405 10013 tablet_bootstrap.cc:654] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:21.724030 9932 tablet_bootstrap.cc:492] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:21.726373 9944 raft_consensus.cc:2243] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:21.726863 9944 raft_consensus.cc:2272] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:21.728562 9944 ts_tablet_manager.cc:1916] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:21.728715 9931 raft_consensus.cc:359] T ab4c185332a24d96bc3d0fddc0d20693 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.729305 9931 raft_consensus.cc:385] T ab4c185332a24d96bc3d0fddc0d20693 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:21.729535 9931 raft_consensus.cc:740] T ab4c185332a24d96bc3d0fddc0d20693 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:21.730048 9931 consensus_queue.cc:260] T ab4c185332a24d96bc3d0fddc0d20693 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.739141 9931 ts_tablet_manager.cc:1434] T ab4c185332a24d96bc3d0fddc0d20693 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260504 13:56:21.739988 9931 tablet_bootstrap.cc:492] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:21.697139 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 07a428049fdb4a68aae729400e2d9674 (table default.Table_0082bd435ced4fdcaeafe8848d05f9bf [id=386315c5331647659a31dc8a2436d7ce]) successfully deleted
W20260504 13:56:21.743098 9329 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 45973us
W20260504 13:56:21.702557 9343 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1fbe09b4078c4eebb52d7d00bc87371c consensus state has no leader
I20260504 13:56:21.745391 9339 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet b37a738f2902402ca76e4edebcdba474 (table default.Table_91cc6c38c9ff43dca25814664068b1c6 [id=da5c427e75e74ea1ae3a5961ed27e0de]). Expected version 1 got 0
I20260504 13:56:21.750416 9931 tablet_bootstrap.cc:654] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:21.750602 9336 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 48b193e958b4414aad29dd2bcee1a61b (table default.Table_2c39b14515bb46f6a0b0f1252d8e273d [id=e9e56499fd2243919308927b3614ae8e]). Expected version 1 got 0
I20260504 13:56:21.751364 9336 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]). Expected version 1 got 0
I20260504 13:56:21.753013 9344 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: default.Table_0d4595642a604b8f8f57d3eadbbacacb [id=666099fdb1a94091ac6fd99cf58f18e2] alter complete (version 1)
I20260504 13:56:21.756655 10393 raft_consensus.cc:493] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:21.757237 10393 raft_consensus.cc:515] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:21.759886 10393 leader_election.cc:290] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:21.762686 9931 tablet_bootstrap.cc:492] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:21.763059 9931 ts_tablet_manager.cc:1403] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.002s
W20260504 13:56:21.767342 9344 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 30) took 2851 ms (client timeout 1000 ms). Trace:
I20260504 13:56:21.767910 9931 raft_consensus.cc:359] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.768496 9931 raft_consensus.cc:385] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:21.768715 9931 raft_consensus.cc:740] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
W20260504 13:56:21.767649 9344 rpcz_store.cc:269] 0504 13:56:18.916301 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:18.943894 (+ 27593us) service_pool.cc:224] Handling call
0504 13:56:21.767277 (+2823383us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:21.182529 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:21.182899 (+ 370us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:21.182933 (+ 34us) write_op.cc:435] Acquired schema lock
0504 13:56:21.182947 (+ 14us) tablet.cc:662] Decoding operations
0504 13:56:21.188912 (+ 5965us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:21.188953 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:21.188987 (+ 34us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:21.189120 (+ 133us) tablet.cc:701] Row locks acquired
0504 13:56:21.189132 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:21.189238 (+ 106us) write_op.cc:270] Start()
0504 13:56:21.189319 (+ 81us) write_op.cc:276] Timestamp: P: 1777902981189179 usec, L: 0
0504 13:56:21.189335 (+ 16us) op_driver.cc:348] REPLICATION: starting
0504 13:56:21.189776 (+ 441us) log.cc:844] Serialized 356 byte log entry
0504 13:56:21.204414 (+ 14638us) op_driver.cc:464] REPLICATION: finished
0504 13:56:21.204809 (+ 395us) write_op.cc:301] APPLY: starting
0504 13:56:21.204873 (+ 64us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:21.204992 (+ 119us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:21.205096 (+ 104us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:21.205241 (+ 145us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:21.205365 (+ 124us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:21.205407 (+ 42us) write_op.cc:312] APPLY: finished
0504 13:56:21.205577 (+ 170us) log.cc:844] Serialized 23 byte log entry
0504 13:56:21.205709 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:21.205767 (+ 58us) write_op.cc:454] Released schema lock
0504 13:56:21.205797 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:21.215616 (+ 9819us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:21.215651 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5810,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":239,"apply.run_cpu_time_us":1179,"apply.run_wall_time_us":1179,"num_ops":1,"prepare.queue_time_us":232552,"prepare.run_cpu_time_us":1948,"prepare.run_wall_time_us":33058,"raft.queue_time_us":1885,"raft.run_cpu_time_us":588,"raft.run_wall_time_us":1310,"replication_time_us":14998}]]}
I20260504 13:56:21.771450 9931 consensus_queue.cc:260] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.772891 9657 tablet_service.cc:1558] Processing DeleteTablet for tablet 0867df9ea115418193b6d9d3fc251923 with delete_type TABLET_DATA_DELETED (replaced by tablet c34877a81d504f46a2f680c5d5ddb4f4) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:21.773228 9659 tablet_service.cc:1558] Processing DeleteTablet for tablet b3793fe8f1ec43ad809403c4593f5ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:12 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:21.773810 9656 tablet_service.cc:1558] Processing DeleteTablet for tablet c34877a81d504f46a2f680c5d5ddb4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 359b0fa01a1648a7bef6f97960a7b9a8) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:21.774448 9932 tablet_bootstrap.cc:654] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:21.776965 9814 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b50573310a04f3694eeab2fef0fb540 with delete_type TABLET_DATA_DELETED (replaced by tablet 8bb77106b48a45a2aaa938dc474fa8bd) from {username='slave'} at 127.0.0.1:56380
W20260504 13:56:21.777581 9337 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 31) took 1825 ms (client timeout 1000 ms). Trace:
I20260504 13:56:21.778393 9814 tablet_service.cc:1558] Processing DeleteTablet for tablet 65052a5c28c147a385144317e21eea41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:21.779431 9656 tablet_service.cc:1558] Processing DeleteTablet for tablet eb7f58645c76441589368072274ebc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:18 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:21.780371 9656 tablet_service.cc:1558] Processing DeleteTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:16 UTC) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:21.780851 9342 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:55533 (request call id 38) took 1412 ms (client timeout 1000 ms). Trace:
I20260504 13:56:21.779557 9931 ts_tablet_manager.cc:1434] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260504 13:56:21.780917 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet 07a428049fdb4a68aae729400e2d9674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:14 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:21.780062 9814 tablet_service.cc:1558] Processing DeleteTablet for tablet 3c43ff1d759a40d68344f25f066416ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:17 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:21.784173 9931 tablet_bootstrap.cc:492] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:21.787451 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "604f1ba5f71041a7877daff8972a994a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:21.788693 9620 leader_election.cc:343] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:21.781172 9342 rpcz_store.cc:269] 0504 13:56:20.367965 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:20.446131 (+ 78166us) service_pool.cc:224] Handling call
0504 13:56:20.460723 (+ 14592us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:21.780793 (+1320070us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":10,"spinlock_wait_cycles":27503488}
W20260504 13:56:21.781049 9337 rpcz_store.cc:269] 0504 13:56:19.952644 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:19.954188 (+ 1544us) service_pool.cc:224] Handling call
0504 13:56:21.777524 (+1823336us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:20.552606 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:20.552957 (+ 351us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:20.552978 (+ 21us) write_op.cc:435] Acquired schema lock
0504 13:56:20.552989 (+ 11us) tablet.cc:662] Decoding operations
0504 13:56:20.558778 (+ 5789us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:20.558838 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:20.558871 (+ 33us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:20.559020 (+ 149us) tablet.cc:701] Row locks acquired
0504 13:56:20.559032 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:20.559122 (+ 90us) write_op.cc:270] Start()
0504 13:56:20.560219 (+ 1097us) write_op.cc:276] Timestamp: P: 1777902980559069 usec, L: 0
0504 13:56:20.560242 (+ 23us) op_driver.cc:348] REPLICATION: starting
0504 13:56:20.560693 (+ 451us) log.cc:844] Serialized 533 byte log entry
0504 13:56:20.569857 (+ 9164us) op_driver.cc:464] REPLICATION: finished
0504 13:56:20.570163 (+ 306us) write_op.cc:301] APPLY: starting
0504 13:56:20.570212 (+ 49us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:20.570302 (+ 90us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:20.570391 (+ 89us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:20.570534 (+ 143us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:20.570669 (+ 135us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:20.570722 (+ 53us) write_op.cc:312] APPLY: finished
0504 13:56:20.570896 (+ 174us) log.cc:844] Serialized 23 byte log entry
0504 13:56:20.571029 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:20.571090 (+ 61us) write_op.cc:454] Released schema lock
0504 13:56:20.571119 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:20.595057 (+ 23938us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:20.595081 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":14478,"reported_tablets":14,"child_traces":[["op",{"apply.queue_time_us":187,"apply.run_cpu_time_us":1154,"apply.run_wall_time_us":1154,"num_ops":1,"prepare.queue_time_us":329884,"prepare.run_cpu_time_us":1818,"prepare.run_wall_time_us":42441,"raft.queue_time_us":210,"raft.run_cpu_time_us":550,"raft.run_wall_time_us":550,"replication_time_us":9539}]]}
I20260504 13:56:21.793771 9931 tablet_bootstrap.cc:654] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:21.795799 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34630d55f4a4e19b33ddf4c13995a5b" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:21.796475 9684 raft_consensus.cc:2468] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:21.819110 9932 tablet_bootstrap.cc:492] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:21.819638 9932 ts_tablet_manager.cc:1403] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.096s user 0.009s sys 0.002s
I20260504 13:56:21.822777 9932 raft_consensus.cc:359] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.823552 9932 raft_consensus.cc:385] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:21.819144 9944 ts_tablet_manager.cc:1929] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:21.824331 9944 log.cc:1199] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/afee197774db411d97468913a207703a
I20260504 13:56:21.825085 9944 ts_tablet_manager.cc:1950] T afee197774db411d97468913a207703a P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:21.828301 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 115) took 2447 ms. Trace:
I20260504 13:56:21.828682 9944 rpcz_store.cc:276] 0504 13:56:19.380902 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:19.392250 (+ 11348us) service_pool.cc:224] Handling call
0504 13:56:21.694750 (+2302500us) ts_tablet_manager.cc:1151] Deleting tablet afee197774db411d97468913a207703a
0504 13:56:21.694787 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:21.807528 (+112741us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.819035 (+ 11507us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.828239 (+ 9204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7171,"tablet-delete.queue_time_us":2274593}
I20260504 13:56:21.757500 10358 raft_consensus.cc:515] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.832691 10358 leader_election.cc:290] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:21.834034 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet afee197774db411d97468913a207703a (table default.Table_91c2d3ed5c5c42758c87f1f726af88c0 [id=a2697dba5f38435c81dc52316712917d]) successfully deleted
I20260504 13:56:21.835011 9944 tablet_replica.cc:333] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:21.835776 9944 raft_consensus.cc:2243] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:21.836280 9944 raft_consensus.cc:2272] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:21.823856 9932 raft_consensus.cc:740] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:21.837865 9932 consensus_queue.cc:260] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.838380 10003 ts_tablet_manager.cc:1929] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260504 13:56:21.838800 10003 log.cc:1199] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/1cf2b59d13da40a2b2160ebdc0b2eb80
I20260504 13:56:21.839569 10003 ts_tablet_manager.cc:1950] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:21.849359 10003 tablet_replica.cc:333] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:21.849350 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 (table default.Table_0eae04160cf241df95cc53da6f4dfacf [id=3822050e2a9d4a9db8b0e36430fd0b7f]) successfully deleted
W20260504 13:56:21.849995 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 2b50573310a04f3694eeab2fef0fb540 because the tablet was not found. No further retry: Not found: Tablet not found: 2b50573310a04f3694eeab2fef0fb540
I20260504 13:56:21.853559 9932 ts_tablet_manager.cc:1434] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.034s user 0.005s sys 0.001s
I20260504 13:56:21.865761 9932 tablet_bootstrap.cc:492] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:21.854540 9944 ts_tablet_manager.cc:1916] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:21.879935 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4c185332a24d96bc3d0fddc0d20693" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:21.880530 9685 raft_consensus.cc:2468] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:21.883270 10003 raft_consensus.cc:2243] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:21.883852 10003 raft_consensus.cc:2272] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:21.886451 9932 tablet_bootstrap.cc:654] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:21.890933 9509 tablet_service.cc:1558] Processing DeleteTablet for tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 with delete_type TABLET_DATA_DELETED (replaced by tablet a862a43c5ac443d9bac98e0472ca23b2) from {username='slave'} at 127.0.0.1:52642
W20260504 13:56:21.900506 9473 consensus_peers.cc:597] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65052a5c28c147a385144317e21eea41. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:21.901327 9473 leader_election.cc:304] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:21.902545 10358 raft_consensus.cc:2804] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:21.902999 10358 raft_consensus.cc:493] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:21.903828 10358 raft_consensus.cc:3060] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:21.905467 9932 tablet_bootstrap.cc:492] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:21.905915 10003 ts_tablet_manager.cc:1916] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:21.905864 9932 ts_tablet_manager.cc:1403] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.002s
I20260504 13:56:21.910535 10358 raft_consensus.cc:515] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.910876 9932 raft_consensus.cc:359] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.911478 9932 raft_consensus.cc:385] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:21.911712 9932 raft_consensus.cc:740] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:21.912750 9944 ts_tablet_manager.cc:1929] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:21.913563 9932 consensus_queue.cc:260] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.913107 9944 log.cc:1199] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/2b50573310a04f3694eeab2fef0fb540
I20260504 13:56:21.916301 9944 ts_tablet_manager.cc:1950] T 2b50573310a04f3694eeab2fef0fb540 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:21.917110 9834 raft_consensus.cc:3060] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:21.924312 9932 ts_tablet_manager.cc:1434] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260504 13:56:21.926422 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4c185332a24d96bc3d0fddc0d20693" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:21.927120 9830 raft_consensus.cc:2468] T ab4c185332a24d96bc3d0fddc0d20693 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:21.928656 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 121) took 1586 ms. Trace:
I20260504 13:56:21.928717 10382 raft_consensus.cc:493] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:21.928963 9944 rpcz_store.cc:276] 0504 13:56:20.342739 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:20.343485 (+ 746us) service_pool.cc:224] Handling call
0504 13:56:21.834860 (+1491375us) ts_tablet_manager.cc:1151] Deleting tablet 2b50573310a04f3694eeab2fef0fb540
0504 13:56:21.834899 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:21.889110 (+ 54211us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.912645 (+ 23535us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.928601 (+ 15956us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10514,"tablet-delete.queue_time_us":1490863}
I20260504 13:56:21.929127 10382 raft_consensus.cc:515] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.930007 9944 tablet_replica.cc:333] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:21.930815 10382 leader_election.cc:290] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:21.930847 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 2b50573310a04f3694eeab2fef0fb540 (table default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]) successfully deleted
I20260504 13:56:21.931914 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be795f04dd54cf19869c937b91d0ddf" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:21.932412 9684 raft_consensus.cc:2468] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:21.925143 9932 tablet_bootstrap.cc:492] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:21.937455 9768 leader_election.cc:304] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:21.938154 10382 raft_consensus.cc:2804] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 13:56:21.938438 10382 raft_consensus.cc:493] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:21.938726 10382 raft_consensus.cc:3060] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Advancing to term 2
I20260504 13:56:21.944790 9944 raft_consensus.cc:2243] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:21.945253 9944 raft_consensus.cc:2272] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:21.945798 10382 raft_consensus.cc:515] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.946904 9944 ts_tablet_manager.cc:1916] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:21.949445 9932 tablet_bootstrap.cc:654] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:21.950538 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be795f04dd54cf19869c937b91d0ddf" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:21.951015 9684 raft_consensus.cc:3060] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Advancing to term 2
I20260504 13:56:21.955304 10382 leader_election.cc:290] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:21.958498 9684 raft_consensus.cc:2468] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 2.
I20260504 13:56:21.959581 9768 leader_election.cc:304] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:21.987394 10393 raft_consensus.cc:493] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:21.987661 9944 ts_tablet_manager.cc:1929] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:21.987983 9944 log.cc:1199] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/50e129980cd84f6ca22cbf2addb06230
I20260504 13:56:21.988564 9944 ts_tablet_manager.cc:1950] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:21.987834 10393 raft_consensus.cc:515] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.989593 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 122) took 1540 ms. Trace:
I20260504 13:56:21.990022 10393 leader_election.cc:290] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:21.991585 9932 tablet_bootstrap.cc:492] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:21.991967 9932 ts_tablet_manager.cc:1403] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.067s user 0.012s sys 0.002s
I20260504 13:56:21.992450 10003 ts_tablet_manager.cc:1929] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:21.992842 10003 log.cc:1199] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/65052a5c28c147a385144317e21eea41
I20260504 13:56:21.993577 10003 ts_tablet_manager.cc:1950] T 65052a5c28c147a385144317e21eea41 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:21.994486 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1cf52a9e58744a99dae1958d2c1f867" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:21.994812 9932 raft_consensus.cc:359] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:21.995412 9932 raft_consensus.cc:385] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:21.995663 9932 raft_consensus.cc:740] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
W20260504 13:56:21.995725 9620 leader_election.cc:343] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:21.996264 9932 consensus_queue.cc:260] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:22.001353 9338 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:38691 (request call id 37) took 1895 ms (client timeout 1000 ms). Trace:
I20260504 13:56:21.989868 9944 rpcz_store.cc:276] 0504 13:56:20.449206 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:20.449322 (+ 116us) service_pool.cc:224] Handling call
0504 13:56:21.929856 (+1480534us) ts_tablet_manager.cc:1151] Deleting tablet 50e129980cd84f6ca22cbf2addb06230
0504 13:56:21.929897 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:21.967089 (+ 37192us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.987549 (+ 20460us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:21.989542 (+ 1993us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1479989}
I20260504 13:56:22.004069 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 50e129980cd84f6ca22cbf2addb06230 (table default.Table_2717e0b44dd14ed1b25e45e06bc0ae88 [id=3e8d8cb6a0b049f8843286a4421e121a]) successfully deleted
I20260504 13:56:22.008114 10393 raft_consensus.cc:493] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:22.008524 10393 raft_consensus.cc:515] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.010476 10393 leader_election.cc:290] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:22.011262 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afad9329cf7745ba8168b2a93f998295" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:22.012307 9620 leader_election.cc:343] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.012677 9944 tablet_replica.cc:333] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:22.013268 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 65052a5c28c147a385144317e21eea41 (table default.Table_038807318bcc4110b77de9582f6c42f5 [id=c19c0f3f26934ad6bbf2d4202cdaa542]) successfully deleted
I20260504 13:56:22.012347 9932 ts_tablet_manager.cc:1434] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260504 13:56:22.014107 9944 raft_consensus.cc:2243] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:22.017741 9932 tablet_bootstrap.cc:492] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:22.014200 10003 tablet_replica.cc:333] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:22.018581 10003 raft_consensus.cc:2243] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:22.019081 10003 raft_consensus.cc:2272] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:22.019310 9944 raft_consensus.cc:2272] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:22.020735 10003 ts_tablet_manager.cc:1916] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:22.021010 9944 ts_tablet_manager.cc:1916] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:22.033452 9932 tablet_bootstrap.cc:654] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:22.045941 10003 ts_tablet_manager.cc:1929] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260504 13:56:22.050707 9328 consensus_peers.cc:597] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 -> Peer 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337): Couldn't send request to peer 6caedf407d13459e8fd477e8b60b7dc7. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260504 13:56:22.057328 9932 tablet_bootstrap.cc:492] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:22.057823 9932 ts_tablet_manager.cc:1403] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.000s
I20260504 13:56:22.059338 10003 log.cc:1199] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/07a428049fdb4a68aae729400e2d9674
I20260504 13:56:22.059666 10382 raft_consensus.cc:2804] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 13:56:22.059968 10003 ts_tablet_manager.cc:1950] T 07a428049fdb4a68aae729400e2d9674 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:22.061838 9932 raft_consensus.cc:359] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.062474 9932 raft_consensus.cc:385] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:22.062588 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 07a428049fdb4a68aae729400e2d9674 (table default.Table_0082bd435ced4fdcaeafe8848d05f9bf [id=386315c5331647659a31dc8a2436d7ce]) successfully deleted
I20260504 13:56:22.062724 9932 raft_consensus.cc:740] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:22.063602 10003 tablet_replica.cc:333] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:22.064749 10003 raft_consensus.cc:2243] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:22.065229 10003 raft_consensus.cc:2272] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:22.067003 9944 ts_tablet_manager.cc:1929] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:22.067402 9944 log.cc:1199] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/c34630d55f4a4e19b33ddf4c13995a5b
I20260504 13:56:22.068044 9944 ts_tablet_manager.cc:1950] T c34630d55f4a4e19b33ddf4c13995a5b P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:22.068014 10003 ts_tablet_manager.cc:1916] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:22.069995 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 124) took 1579 ms. Trace:
I20260504 13:56:22.070282 9944 rpcz_store.cc:276] 0504 13:56:20.491003 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:20.517623 (+ 26620us) service_pool.cc:224] Handling call
0504 13:56:22.012534 (+1494911us) ts_tablet_manager.cc:1151] Deleting tablet c34630d55f4a4e19b33ddf4c13995a5b
0504 13:56:22.012576 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:22.052135 (+ 39559us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:22.066882 (+ 14747us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:22.069933 (+ 3051us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1485636}
I20260504 13:56:22.071455 9932 consensus_queue.cc:260] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.088693 9932 ts_tablet_manager.cc:1434] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.031s user 0.005s sys 0.000s
I20260504 13:56:22.091708 9932 tablet_bootstrap.cc:492] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:22.092767 10003 ts_tablet_manager.cc:1929] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:22.093927 10003 log.cc:1199] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/3c43ff1d759a40d68344f25f066416ac
I20260504 13:56:22.094672 10003 ts_tablet_manager.cc:1950] T 3c43ff1d759a40d68344f25f066416ac P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:22.096956 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet c34630d55f4a4e19b33ddf4c13995a5b (table default.Table_434bdc9867ad475a9719ba0c3760ef71 [id=db5cbb438d8f4da981e70b12713134f1]) successfully deleted
I20260504 13:56:22.098009 9944 tablet_replica.cc:333] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:22.099565 9944 raft_consensus.cc:2243] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260504 13:56:22.105036 9339 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 32) took 1102 ms (client timeout 1000 ms). Trace:
W20260504 13:56:22.105367 9339 rpcz_store.cc:269] 0504 13:56:21.002665 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.003858 (+ 1193us) service_pool.cc:224] Handling call
0504 13:56:21.012653 (+ 8795us) spinlock_profiling.cc:243] Waited 5.36 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:22.104904 (+1092251us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87586,"reported_tablets":22,"spinlock_wait_cycles":12317440}
I20260504 13:56:22.106583 10455 raft_consensus.cc:493] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:22.106364 9656 tablet_service.cc:1558] Processing DeleteTablet for tablet 65052a5c28c147a385144317e21eea41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:22.107573 9944 raft_consensus.cc:2272] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:22.107327 10455 raft_consensus.cc:515] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:22.102974 10382 raft_consensus.cc:697] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [term 2 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:22.111792 10382 consensus_queue.cc:237] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.113955 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34877a81d504f46a2f680c5d5ddb4f4" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:22.114274 9618 leader_election.cc:304] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:22.114640 9684 raft_consensus.cc:2468] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:22.115334 9618 leader_election.cc:304] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:22.116569 10393 raft_consensus.cc:2804] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:22.116870 10393 raft_consensus.cc:493] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:22.117174 10393 raft_consensus.cc:3060] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:22.117036 9768 leader_election.cc:304] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:22.117465 10455 leader_election.cc:290] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:22.119896 10455 raft_consensus.cc:2804] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 13:56:22.120293 10455 raft_consensus.cc:493] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:22.121677 10455 raft_consensus.cc:3060] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Advancing to term 2
I20260504 13:56:22.125617 10393 raft_consensus.cc:515] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:22.001710 9338 rpcz_store.cc:269] 0504 13:56:20.105699 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:20.206691 (+100992us) service_pool.cc:224] Handling call
0504 13:56:22.001284 (+1794593us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:21.215943 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:21.216424 (+ 481us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:21.216455 (+ 31us) write_op.cc:435] Acquired schema lock
0504 13:56:21.216475 (+ 20us) tablet.cc:662] Decoding operations
0504 13:56:21.216953 (+ 478us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:21.216996 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:21.217015 (+ 19us) tablet.cc:685] Acquiring locks for 3 operations
0504 13:56:21.217242 (+ 227us) tablet.cc:701] Row locks acquired
0504 13:56:21.217255 (+ 13us) write_op.cc:260] PREPARE: finished
0504 13:56:21.217343 (+ 88us) write_op.cc:270] Start()
0504 13:56:21.217414 (+ 71us) write_op.cc:276] Timestamp: P: 1777902981217285 usec, L: 0
0504 13:56:21.217427 (+ 13us) op_driver.cc:348] REPLICATION: starting
0504 13:56:21.217877 (+ 450us) log.cc:844] Serialized 1293 byte log entry
0504 13:56:21.636563 (+418686us) op_driver.cc:464] REPLICATION: finished
0504 13:56:21.636779 (+ 216us) write_op.cc:301] APPLY: starting
0504 13:56:21.636830 (+ 51us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:21.636961 (+ 131us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:21.637052 (+ 91us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:21.637409 (+ 357us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:21.637551 (+ 142us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0504 13:56:21.637590 (+ 39us) write_op.cc:312] APPLY: finished
0504 13:56:21.637774 (+ 184us) log.cc:844] Serialized 35 byte log entry
0504 13:56:21.637937 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:21.638005 (+ 68us) write_op.cc:454] Released schema lock
0504 13:56:21.638035 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":108608,"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":82,"apply.run_cpu_time_us":1443,"apply.run_wall_time_us":1443,"num_ops":3,"prepare.queue_time_us":253947,"prepare.run_cpu_time_us":2272,"prepare.run_wall_time_us":186209,"raft.queue_time_us":79,"raft.run_cpu_time_us":700,"raft.run_wall_time_us":701,"replication_time_us":419035}]]}
I20260504 13:56:22.130816 9944 ts_tablet_manager.cc:1916] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:22.135530 9932 tablet_bootstrap.cc:654] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:22.138281 10455 raft_consensus.cc:515] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:22.138891 10393 leader_election.cc:290] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:22.139812 10393 raft_consensus.cc:2804] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:22.140128 10393 raft_consensus.cc:493] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:22.140429 10393 raft_consensus.cc:3060] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:22.146090 9618 leader_election.cc:343] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.147784 9343 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_2c39b14515bb46f6a0b0f1252d8e273d" } modify_external_catalogs: true
I20260504 13:56:22.148277 9343 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_2c39b14515bb46f6a0b0f1252d8e273d" } modify_external_catalogs: true
I20260504 13:56:22.150063 9335 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35768 (request call id 209) took 1315 ms. Trace:
I20260504 13:56:22.151355 9337 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_0d2ce6c9ff35497b82dd58da3f913ecb" } modify_external_catalogs: true
I20260504 13:56:22.151808 9337 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_0d2ce6c9ff35497b82dd58da3f913ecb" } modify_external_catalogs: true
I20260504 13:56:22.153329 9618 leader_election.cc:304] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:22.153467 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34877a81d504f46a2f680c5d5ddb4f4" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:22.150346 9335 rpcz_store.cc:276] 0504 13:56:20.834801 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:20.944369 (+109568us) service_pool.cc:224] Handling call
0504 13:56:20.981727 (+ 37358us) spinlock_profiling.cc:243] Waited 36.6 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b0936 00007fc27b2bd524 00007fc27b2bd4b2 00007fc27b2bd3e1 00007fc27b2bd35c 00007fc27b2bc612 00007fc27958bbcc 00007fc27958b006
0504 13:56:20.985986 (+ 4259us) catalog_manager.cc:2101] Acquired catalog manager lock
0504 13:56:20.987527 (+ 1541us) catalog_manager.cc:2147] Created new table and tablet info
0504 13:56:22.149497 (+1161970us) catalog_manager.cc:2207] Wrote table and tablets to system table
0504 13:56:22.149753 (+ 256us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0504 13:56:22.150010 (+ 257us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:21.727377 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:21.727740 (+ 363us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:21.727763 (+ 23us) write_op.cc:435] Acquired schema lock
0504 13:56:21.727774 (+ 11us) tablet.cc:662] Decoding operations
0504 13:56:21.728014 (+ 240us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:21.728050 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:21.728072 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:21.728269 (+ 197us) tablet.cc:701] Row locks acquired
0504 13:56:21.728280 (+ 11us) write_op.cc:260] PREPARE: finished
0504 13:56:21.728381 (+ 101us) write_op.cc:270] Start()
0504 13:56:21.728455 (+ 74us) write_op.cc:276] Timestamp: P: 1777902981728300 usec, L: 0
0504 13:56:21.728470 (+ 15us) op_driver.cc:348] REPLICATION: starting
0504 13:56:21.728983 (+ 513us) log.cc:844] Serialized 433 byte log entry
0504 13:56:22.147459 (+418476us) op_driver.cc:464] REPLICATION: finished
0504 13:56:22.147826 (+ 367us) write_op.cc:301] APPLY: starting
0504 13:56:22.147877 (+ 51us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:22.147973 (+ 96us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:22.148066 (+ 93us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:22.148458 (+ 392us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:22.148585 (+ 127us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0504 13:56:22.148629 (+ 44us) write_op.cc:312] APPLY: finished
0504 13:56:22.148794 (+ 165us) log.cc:844] Serialized 29 byte log entry
0504 13:56:22.148927 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:22.148990 (+ 63us) write_op.cc:454] Released schema lock
0504 13:56:22.149024 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:22.149238 (+ 214us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:22.149272 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":84128128,"child_traces":[["op",{"apply.queue_time_us":233,"apply.run_cpu_time_us":1382,"apply.run_wall_time_us":1382,"num_ops":2,"prepare.queue_time_us":738803,"prepare.run_cpu_time_us":1745,"prepare.run_wall_time_us":2847,"replication_time_us":418892}]]}
I20260504 13:56:22.153965 9684 raft_consensus.cc:3060] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Advancing to term 2
W20260504 13:56:22.154316 9618 leader_election.cc:343] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.154668 9618 leader_election.cc:304] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:22.155143 9618 raft_consensus.cc:2686] T 50e129980cd84f6ca22cbf2addb06230 P 5beceb6317444247b31b44a6581707c6: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260504 13:56:22.157162 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030b35c3409143108c34ab1f0ee1b96d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:22.158252 9620 leader_election.cc:343] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.158865 10457 raft_consensus.cc:2749] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:22.159718 10382 raft_consensus.cc:493] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:22.160130 10382 raft_consensus.cc:515] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:22.160653 9932 tablet_bootstrap.cc:492] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:22.161020 9932 ts_tablet_manager.cc:1403] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.069s user 0.008s sys 0.000s
I20260504 13:56:22.163084 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e4239942f34fb081a026a35740e465" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:22.099504 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 3c43ff1d759a40d68344f25f066416ac (table default.Table_1f5a66e30f35482dab7c556daebc87c9 [id=c268fa33d24540b4978e3feebb0f793f]) successfully deleted
W20260504 13:56:22.164266 9330 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 64777.6us
I20260504 13:56:22.167730 9814 tablet_service.cc:1558] Processing DeleteTablet for tablet 65052a5c28c147a385144317e21eea41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:22.168443 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet 07a428049fdb4a68aae729400e2d9674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:14 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:22.169157 9807 tablet_service.cc:1558] Processing DeleteTablet for tablet 3c43ff1d759a40d68344f25f066416ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:17 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:22.169672 9344 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_2717e0b44dd14ed1b25e45e06bc0ae88" } new_table_name: "default.Table_270726cef63b42e39368cdf6aae3fe18" modify_external_catalogs: true
I20260504 13:56:22.169811 9806 tablet_service.cc:1558] Processing DeleteTablet for tablet 70caac2e256d419dbc2e70cb5dfffe97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:18 UTC) from {username='slave'} at 127.0.0.1:56380
W20260504 13:56:22.173703 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 65052a5c28c147a385144317e21eea41 because the tablet was not found. No further retry: Not found: Tablet not found: 65052a5c28c147a385144317e21eea41
I20260504 13:56:22.174160 9341 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35768 (request call id 212) took 1137 ms. Trace:
W20260504 13:56:22.174285 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 07a428049fdb4a68aae729400e2d9674 because the tablet was not found. No further retry: Not found: Tablet not found: 07a428049fdb4a68aae729400e2d9674
W20260504 13:56:22.174782 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 3c43ff1d759a40d68344f25f066416ac because the tablet was not found. No further retry: Not found: Tablet not found: 3c43ff1d759a40d68344f25f066416ac
W20260504 13:56:22.175333 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 70caac2e256d419dbc2e70cb5dfffe97 because the tablet was not found. No further retry: Not found: Tablet not found: 70caac2e256d419dbc2e70cb5dfffe97
I20260504 13:56:22.174481 9341 rpcz_store.cc:276] 0504 13:56:21.036552 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.090772 (+ 54220us) service_pool.cc:224] Handling call
0504 13:56:21.105327 (+ 14555us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b0936 00007fc27b2bd524 00007fc27b2bd4b2 00007fc27b2bd3e1 00007fc27b2bd35c 00007fc27b2bc612 00007fc27958bbcc 00007fc27958b006
0504 13:56:21.107735 (+ 2408us) catalog_manager.cc:2101] Acquired catalog manager lock
0504 13:56:21.109110 (+ 1375us) catalog_manager.cc:2147] Created new table and tablet info
0504 13:56:22.173598 (+1064488us) catalog_manager.cc:2207] Wrote table and tablets to system table
0504 13:56:22.173852 (+ 254us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0504 13:56:22.174110 (+ 258us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:21.730328 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:21.730761 (+ 433us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:21.730783 (+ 22us) write_op.cc:435] Acquired schema lock
0504 13:56:21.730796 (+ 13us) tablet.cc:662] Decoding operations
0504 13:56:21.731063 (+ 267us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:21.731094 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:21.731123 (+ 29us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:21.731348 (+ 225us) tablet.cc:701] Row locks acquired
0504 13:56:21.731360 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:21.731451 (+ 91us) write_op.cc:270] Start()
0504 13:56:21.731530 (+ 79us) write_op.cc:276] Timestamp: P: 1777902981731369 usec, L: 0
0504 13:56:21.731549 (+ 19us) op_driver.cc:348] REPLICATION: starting
0504 13:56:22.145266 (+413717us) log.cc:844] Serialized 433 byte log entry
0504 13:56:22.171548 (+ 26282us) op_driver.cc:464] REPLICATION: finished
0504 13:56:22.171874 (+ 326us) write_op.cc:301] APPLY: starting
0504 13:56:22.171929 (+ 55us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:22.172026 (+ 97us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:22.172113 (+ 87us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:22.172544 (+ 431us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:22.172677 (+ 133us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0504 13:56:22.172720 (+ 43us) write_op.cc:312] APPLY: finished
0504 13:56:22.172884 (+ 164us) log.cc:844] Serialized 29 byte log entry
0504 13:56:22.173023 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:22.173085 (+ 62us) write_op.cc:454] Released schema lock
0504 13:56:22.173113 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:22.173342 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:22.173376 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":30658944,"child_traces":[["op",{"apply.queue_time_us":185,"apply.run_cpu_time_us":1432,"apply.run_wall_time_us":1431,"num_ops":2,"prepare.queue_time_us":620130,"prepare.run_cpu_time_us":2152,"prepare.run_wall_time_us":435960,"raft.queue_time_us":147,"raft.run_cpu_time_us":591,"raft.run_wall_time_us":592,"replication_time_us":439902}]]}
I20260504 13:56:22.178887 10382 leader_election.cc:290] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:21.914824 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70caac2e256d419dbc2e70cb5dfffe97" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:22.181279 9684 raft_consensus.cc:2468] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 2.
I20260504 13:56:22.181670 9340 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35768 (request call id 213) took 1051 ms. Trace:
I20260504 13:56:21.924649 9473 leader_election.cc:304] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
W20260504 13:56:22.182335 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 257816us
I20260504 13:56:22.182581 9768 leader_election.cc:304] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:22.183466 10318 raft_consensus.cc:2804] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:22.183537 10382 raft_consensus.cc:2804] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 13:56:22.183786 10318 raft_consensus.cc:493] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:22.184095 10318 raft_consensus.cc:3060] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:22.183962 9335 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_203b5e06868048d2a803bf466f3959d3" } new_table_name: "default.Table_0870a7a3564d42cb8531cae341a9f582" modify_external_catalogs: true
I20260504 13:56:22.184118 10382 raft_consensus.cc:697] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [term 2 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:22.184974 10382 consensus_queue.cc:237] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:22.185245 9932 raft_consensus.cc:359] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.185928 9932 raft_consensus.cc:385] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:22.186192 9932 raft_consensus.cc:740] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:22.188087 10455 leader_election.cc:290] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:22.188000 10393 raft_consensus.cc:515] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.181963 9340 rpcz_store.cc:276] 0504 13:56:21.130391 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.130613 (+ 222us) service_pool.cc:224] Handling call
0504 13:56:21.176055 (+ 45442us) spinlock_profiling.cc:243] Waited 44.7 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b0936 00007fc27b2bd524 00007fc27b2bd4b2 00007fc27b2bd3e1 00007fc27b2bd35c 00007fc27b2bc612 00007fc27958bbcc 00007fc27958b006
0504 13:56:21.178387 (+ 2332us) catalog_manager.cc:2101] Acquired catalog manager lock
0504 13:56:21.180643 (+ 2256us) catalog_manager.cc:2147] Created new table and tablet info
0504 13:56:22.181105 (+1000462us) catalog_manager.cc:2207] Wrote table and tablets to system table
0504 13:56:22.181389 (+ 284us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0504 13:56:22.181619 (+ 230us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:22.166408 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:22.166773 (+ 365us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:22.166795 (+ 22us) write_op.cc:435] Acquired schema lock
0504 13:56:22.166807 (+ 12us) tablet.cc:662] Decoding operations
0504 13:56:22.167063 (+ 256us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:22.167096 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:22.167118 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:22.167318 (+ 200us) tablet.cc:701] Row locks acquired
0504 13:56:22.167330 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:22.167427 (+ 97us) write_op.cc:270] Start()
0504 13:56:22.167500 (+ 73us) write_op.cc:276] Timestamp: P: 1777902982167368 usec, L: 0
0504 13:56:22.167515 (+ 15us) op_driver.cc:348] REPLICATION: starting
0504 13:56:22.167957 (+ 442us) log.cc:844] Serialized 433 byte log entry
0504 13:56:22.178232 (+ 10275us) op_driver.cc:464] REPLICATION: finished
0504 13:56:22.179828 (+ 1596us) write_op.cc:301] APPLY: starting
0504 13:56:22.179875 (+ 47us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:22.179966 (+ 91us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:22.180053 (+ 87us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:22.180394 (+ 341us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:22.180515 (+ 121us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0504 13:56:22.180559 (+ 44us) write_op.cc:312] APPLY: finished
0504 13:56:22.180709 (+ 150us) log.cc:844] Serialized 29 byte log entry
0504 13:56:22.180837 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:22.180899 (+ 62us) write_op.cc:454] Released schema lock
0504 13:56:22.180930 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":102751104,"child_traces":[["op",{"apply.queue_time_us":1462,"apply.run_cpu_time_us":1301,"apply.run_wall_time_us":3482,"num_ops":2,"prepare.queue_time_us":984392,"replication_time_us":10623}]]}
I20260504 13:56:22.191404 9932 consensus_queue.cc:260] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.193109 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_5944297f8c104978bbc6913ecd8f47a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:22.193498 10431 consensus_queue.cc:1048] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260504 13:56:22.194850 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5944297f8c104978bbc6913ecd8f47a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:21.724726 10391 rpcz_store.cc:276] 0504 13:56:20.444823 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:20.444932 (+ 109us) service_pool.cc:224] Handling call
0504 13:56:21.622134 (+1177202us) ts_tablet_manager.cc:1151] Deleting tablet 50e129980cd84f6ca22cbf2addb06230
0504 13:56:21.622191 (+ 57us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:21.622466 (+ 275us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1176670}
I20260504 13:56:22.195569 10391 tablet_replica.cc:333] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:22.198132 10429 consensus_queue.cc:1048] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260504 13:56:22.082144 9474 consensus_peers.cc:597] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70caac2e256d419dbc2e70cb5dfffe97. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:22.205744 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 123696us
I20260504 13:56:22.110323 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "604f1ba5f71041a7877daff8972a994a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:22.112542 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1cf52a9e58744a99dae1958d2c1f867" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:22.113562 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afad9329cf7745ba8168b2a93f998295" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:22.134779 9505 tablet_service.cc:1558] Processing DeleteTablet for tablet 3c43ff1d759a40d68344f25f066416ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:17 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:22.212044 10429 consensus_queue.cc:1048] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:22.215687 10429 consensus_queue.cc:1048] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:22.216786 9474 leader_election.cc:343] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.139631 9510 tablet_service.cc:1558] Processing DeleteTablet for tablet 07a428049fdb4a68aae729400e2d9674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:14 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:22.140396 9506 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b50573310a04f3694eeab2fef0fb540 with delete_type TABLET_DATA_DELETED (replaced by tablet 8bb77106b48a45a2aaa938dc474fa8bd) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:22.141180 9507 tablet_service.cc:1558] Processing DeleteTablet for tablet c34877a81d504f46a2f680c5d5ddb4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 359b0fa01a1648a7bef6f97960a7b9a8) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:22.177471 10013 tablet_bootstrap.cc:492] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:22.225808 10013 ts_tablet_manager.cc:1403] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.698s user 0.011s sys 0.002s
I20260504 13:56:22.228507 10013 raft_consensus.cc:359] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.229066 10013 raft_consensus.cc:385] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:22.229326 10013 raft_consensus.cc:740] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:22.229861 10013 consensus_queue.cc:260] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.198956 9944 ts_tablet_manager.cc:1929] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:22.211510 9684 raft_consensus.cc:1240] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [term 2 FOLLOWER]: Rejecting Update request from peer 7b79fe7dd8934dd996393a2d96eabf00 for earlier term 1. Current term is 2. Ops: []
I20260504 13:56:22.214493 9932 ts_tablet_manager.cc:1434] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
W20260504 13:56:22.082918 9520 consensus_meta.cc:287] T 3404ba04d3ec4a44a5e625536b941b58 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent flushing consensus metadata: real 0.546s user 0.005s sys 0.001s
I20260504 13:56:22.217968 10429 consensus_queue.cc:1048] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:22.234537 10013 ts_tablet_manager.cc:1434] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260504 13:56:22.215353 10393 leader_election.cc:290] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:22.224306 9685 raft_consensus.cc:1240] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [term 2 FOLLOWER]: Rejecting Update request from peer 7b79fe7dd8934dd996393a2d96eabf00 for earlier term 1. Current term is 2. Ops: []
I20260504 13:56:22.241271 9932 tablet_bootstrap.cc:492] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:22.231346 9944 log.cc:1199] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/70caac2e256d419dbc2e70cb5dfffe97
I20260504 13:56:22.253371 9944 ts_tablet_manager.cc:1950] T 70caac2e256d419dbc2e70cb5dfffe97 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:22.254755 10013 tablet_bootstrap.cc:492] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:22.255923 9932 tablet_bootstrap.cc:654] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:22.138675 9508 consensus_meta.cc:287] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent flushing consensus metadata: real 0.521s user 0.004s sys 0.003s
I20260504 13:56:22.265538 10358 leader_election.cc:290] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:22.266358 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 130) took 1325 ms. Trace:
I20260504 13:56:22.266846 9944 rpcz_store.cc:276] 0504 13:56:20.940808 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:20.940927 (+ 119us) service_pool.cc:224] Handling call
0504 13:56:22.097865 (+1156938us) ts_tablet_manager.cc:1151] Deleting tablet 70caac2e256d419dbc2e70cb5dfffe97
0504 13:56:22.097902 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:22.192564 (+ 94662us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:22.198842 (+ 6278us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:22.266306 (+ 67464us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1156441}
I20260504 13:56:22.267954 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 70caac2e256d419dbc2e70cb5dfffe97 (table default.Table_c5584202645f4ef78e4f6e00aa2bf2c1 [id=2665806fb81f4eda8981fede847cbfcf]) successfully deleted
I20260504 13:56:22.268879 9944 tablet_replica.cc:333] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:22.269686 9944 raft_consensus.cc:2243] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:22.270087 9944 raft_consensus.cc:2272] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:22.275817 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 50e129980cd84f6ca22cbf2addb06230 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50e129980cd84f6ca22cbf2addb06230 already in progress: creating tablet
I20260504 13:56:22.280718 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b13303d2a14c4187b82da9f25ce9753e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:22.281842 9620 leader_election.cc:343] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.286996 9932 tablet_bootstrap.cc:492] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:22.287436 9932 ts_tablet_manager.cc:1403] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.002s
I20260504 13:56:22.288383 10465 tablet.cc:1724] T b37a738f2902402ca76e4edebcdba474 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:22.266026 10391 raft_consensus.cc:2243] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:22.268908 10355 consensus_queue.cc:1048] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:22.245955 10318 raft_consensus.cc:515] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.292850 10318 leader_election.cc:290] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:22.322808 9343 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 48b193e958b4414aad29dd2bcee1a61b
I20260504 13:56:22.324225 9343 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35768 (request call id 214) took 1151 ms. Trace:
I20260504 13:56:22.328727 9806 tablet_service.cc:1558] Processing DeleteTablet for tablet 48b193e958b4414aad29dd2bcee1a61b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:22.329622 10003 tablet_replica.cc:333] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:22.330475 10003 raft_consensus.cc:2243] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:22.331027 10003 raft_consensus.cc:2272] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:22.333118 10003 ts_tablet_manager.cc:1916] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:22.324496 9343 rpcz_store.cc:276] 0504 13:56:21.172633 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.744371 (+571738us) service_pool.cc:224] Handling call
0504 13:56:22.147703 (+403332us) spinlock_profiling.cc:243] Waited 403 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b0c46 00007fc27b2c2024 00007fc27b2c1fb2 00007fc27b2c1ee1 00007fc27b2c1e5c 00007fc27b2c1112 00007fc27958bbcc 00007fc27958b006
0504 13:56:22.148628 (+ 925us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:22.149105 (+ 477us) catalog_manager.cc:2815] Modifying in-memory table state
0504 13:56:22.149228 (+ 123us) catalog_manager.cc:2824] Locking tablets
0504 13:56:22.149741 (+ 513us) catalog_manager.cc:2838] Removing table and tablets from system table
0504 13:56:22.322460 (+172719us) catalog_manager.cc:2856] Removing table from by-name map
0504 13:56:22.322643 (+ 183us) catalog_manager.cc:2872] Committing in-memory state
0504 13:56:22.322727 (+ 84us) catalog_manager.cc:2876] Aborting table tasks
0504 13:56:22.324154 (+ 1427us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:22.308301 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:22.308656 (+ 355us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:22.308679 (+ 23us) write_op.cc:435] Acquired schema lock
0504 13:56:22.308693 (+ 14us) tablet.cc:662] Decoding operations
0504 13:56:22.309030 (+ 337us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:22.309070 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:22.309093 (+ 23us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:22.309336 (+ 243us) tablet.cc:701] Row locks acquired
0504 13:56:22.309350 (+ 14us) write_op.cc:260] PREPARE: finished
0504 13:56:22.309447 (+ 97us) write_op.cc:270] Start()
0504 13:56:22.309524 (+ 77us) write_op.cc:276] Timestamp: P: 1777902982309378 usec, L: 0
0504 13:56:22.309539 (+ 15us) op_driver.cc:348] REPLICATION: starting
0504 13:56:22.310003 (+ 464us) log.cc:844] Serialized 796 byte log entry
0504 13:56:22.320269 (+ 10266us) op_driver.cc:464] REPLICATION: finished
0504 13:56:22.320577 (+ 308us) write_op.cc:301] APPLY: starting
0504 13:56:22.320625 (+ 48us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:22.320780 (+ 155us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:22.320864 (+ 84us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:22.321111 (+ 247us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:22.321264 (+ 153us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:22.321305 (+ 41us) write_op.cc:312] APPLY: finished
0504 13:56:22.321461 (+ 156us) log.cc:844] Serialized 29 byte log entry
0504 13:56:22.321581 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:22.322118 (+ 537us) write_op.cc:454] Released schema lock
0504 13:56:22.322148 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":926019456,"child_traces":[["op",{"apply.queue_time_us":195,"apply.run_cpu_time_us":1749,"apply.run_wall_time_us":1749,"num_ops":2,"prepare.queue_time_us":155029,"raft.queue_time_us":743,"raft.run_cpu_time_us":699,"raft.run_wall_time_us":699,"replication_time_us":10634}]]}
I20260504 13:56:22.344224 10454 raft_consensus.cc:493] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:22.344636 10454 raft_consensus.cc:515] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:22.346534 10454 leader_election.cc:290] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:22.360615 10003 ts_tablet_manager.cc:1929] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:22.370537 10003 log.cc:1199] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/48b193e958b4414aad29dd2bcee1a61b
I20260504 13:56:22.371937 10003 ts_tablet_manager.cc:1950] T 48b193e958b4414aad29dd2bcee1a61b P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:22.375614 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 48b193e958b4414aad29dd2bcee1a61b (table default.Table_2c39b14515bb46f6a0b0f1252d8e273d [id=e9e56499fd2243919308927b3614ae8e]) successfully deleted
I20260504 13:56:22.378580 9340 master_service.cc:954] ReplaceTablet: received request to replace tablet 030b35c3409143108c34ab1f0ee1b96d from {username='slave'} at 127.0.0.1:35768
W20260504 13:56:22.386576 9330 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 1fbe09b4078c4eebb52d7d00bc87371c (table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1fbe09b4078c4eebb52d7d00bc87371c consensus state has no leader
W20260504 13:56:22.397372 9931 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.063s sys 0.121s
W20260504 13:56:22.397636 9931 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.602s user 0.063s sys 0.121s
I20260504 13:56:22.399330 9931 tablet_bootstrap.cc:492] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:22.399708 9931 ts_tablet_manager.cc:1403] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.616s user 0.071s sys 0.121s
W20260504 13:56:22.403133 9336 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:55533 (request call id 39) took 791 ms (client timeout 1000 ms). Trace:
W20260504 13:56:22.403482 9336 rpcz_store.cc:269] 0504 13:56:21.374717 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.650915 (+276198us) service_pool.cc:224] Handling call
0504 13:56:21.665665 (+ 14750us) spinlock_profiling.cc:243] Waited 7.75 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:22.166079 (+500414us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100910,"reported_tablets":10,"spinlock_wait_cycles":17830656}
I20260504 13:56:22.403877 9931 raft_consensus.cc:359] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.404454 9931 raft_consensus.cc:385] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:22.407577 9931 raft_consensus.cc:740] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:22.289736 9932 raft_consensus.cc:359] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.408099 9932 raft_consensus.cc:385] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:22.408382 9932 raft_consensus.cc:740] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:22.408138 9931 consensus_queue.cc:260] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.409000 9932 consensus_queue.cc:260] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.410944 9932 ts_tablet_manager.cc:1434] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.123s user 0.003s sys 0.002s
I20260504 13:56:22.290962 9944 ts_tablet_manager.cc:1916] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:22.413244 9931 ts_tablet_manager.cc:1434] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260504 13:56:22.413252 9508 tablet_service.cc:1558] Processing DeleteTablet for tablet 48b193e958b4414aad29dd2bcee1a61b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:22.284492 9518 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:52642 (request call id 132) took 1019 ms. Trace:
I20260504 13:56:22.414660 9518 rpcz_store.cc:276] 0504 13:56:21.265240 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.455577 (+190337us) service_pool.cc:224] Handling call
0504 13:56:21.457572 (+ 1995us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:21.457759 (+ 187us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:21.855468 (+397709us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:22.284025 (+428557us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:22.284437 (+ 412us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:22.415748 9931 tablet_bootstrap.cc:492] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:22.292832 10393 consensus_queue.cc:1048] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:22.331328 9656 tablet_service.cc:1558] Processing DeleteTablet for tablet 48b193e958b4414aad29dd2bcee1a61b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:22.340533 9618 leader_election.cc:343] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.424189 9618 leader_election.cc:304] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:22.424643 9931 tablet_bootstrap.cc:654] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:22.424939 9618 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 84441.5us
W20260504 13:56:22.426568 9618 leader_election.cc:343] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.347980 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1501455a0af348e38746d100987afdd0" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:22.297528 10462 tablet.cc:1724] T b37a738f2902402ca76e4edebcdba474 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260504 13:56:22.338899 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 65416.4us
I20260504 13:56:22.341459 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030b35c3409143108c34ab1f0ee1b96d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:22.342393 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b13303d2a14c4187b82da9f25ce9753e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:22.428014 9539 raft_consensus.cc:3060] T b13303d2a14c4187b82da9f25ce9753e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:22.362251 10391 raft_consensus.cc:2272] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:22.279933 9511 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:52642 (request call id 139) took 1011 ms. Trace:
I20260504 13:56:22.430110 10393 raft_consensus.cc:2749] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:22.426911 9618 leader_election.cc:304] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:22.435812 9618 leader_election.cc:343] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.436154 9618 leader_election.cc:304] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:22.436599 9618 raft_consensus.cc:2686] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260504 13:56:22.437530 10393 raft_consensus.cc:2749] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:22.438949 9932 tablet_bootstrap.cc:492] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:22.439601 9536 raft_consensus.cc:3060] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:22.439292 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_a992fb6e038e4408a781d6d8ead37f54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:22.441131 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a992fb6e038e4408a781d6d8ead37f54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:22.443581 9343 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_2585508fc4214c8b82d2957644309ac2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:22.444981 9343 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2585508fc4214c8b82d2957644309ac2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:22.446663 9834 raft_consensus.cc:3060] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:22.355716 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4c185332a24d96bc3d0fddc0d20693" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:22.447376 9684 raft_consensus.cc:3060] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:22.447715 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_5bfac12ab2f84c06b1f992848d8f5041"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:22.449380 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bfac12ab2f84c06b1f992848d8f5041 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:22.393312 9620 consensus_peers.cc:597] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48b193e958b4414aad29dd2bcee1a61b. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:22.450158 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 57022.9us
I20260504 13:56:22.456352 9767 leader_election.cc:304] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
W20260504 13:56:22.456889 9767 raft_consensus.cc:2686] T 70caac2e256d419dbc2e70cb5dfffe97 P d4051d2bf0bf410eb8acd107560df888: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260504 13:56:22.460654 9829 raft_consensus.cc:1240] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [term 2 LEADER]: Rejecting Update request from peer 7b79fe7dd8934dd996393a2d96eabf00 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260504 13:56:22.461087 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34630d55f4a4e19b33ddf4c13995a5b" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:22.464727 9829 raft_consensus.cc:1240] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [term 2 LEADER]: Rejecting Update request from peer 7b79fe7dd8934dd996393a2d96eabf00 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260504 13:56:22.469758 9684 raft_consensus.cc:2468] T ab4c185332a24d96bc3d0fddc0d20693 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:22.469823 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab4c185332a24d96bc3d0fddc0d20693" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:22.470340 9829 raft_consensus.cc:3060] T ab4c185332a24d96bc3d0fddc0d20693 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:22.476724 9829 raft_consensus.cc:2468] T ab4c185332a24d96bc3d0fddc0d20693 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:22.478430 9931 tablet_bootstrap.cc:492] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:22.478932 9931 ts_tablet_manager.cc:1403] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.063s user 0.019s sys 0.005s
I20260504 13:56:22.479782 9932 tablet_bootstrap.cc:654] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:22.480274 9829 raft_consensus.cc:3060] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:22.481662 9931 raft_consensus.cc:359] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.482278 9931 raft_consensus.cc:385] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:22.482523 9931 raft_consensus.cc:740] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:22.484278 9830 raft_consensus.cc:3060] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:22.483062 9931 consensus_queue.cc:260] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.494081 9931 ts_tablet_manager.cc:1434] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
W20260504 13:56:22.495719 9344 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c47b6e3b4d0c45c28384f7aae6b749de has no consensus state
I20260504 13:56:22.496189 9344 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35768 (request call id 215) took 1323 ms. Trace:
I20260504 13:56:22.497233 9931 tablet_bootstrap.cc:492] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:22.496466 9344 rpcz_store.cc:276] 0504 13:56:21.173046 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.770570 (+597524us) service_pool.cc:224] Handling call
0504 13:56:22.169590 (+399020us) spinlock_profiling.cc:243] Waited 398 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b0fb6 00007fc27b2c6b24 00007fc27b2c6ab2 00007fc27b2c69e1 00007fc27b2c695c 00007fc27b2c5c12 00007fc27958bbcc 00007fc27958b006
0504 13:56:22.170223 (+ 633us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:22.170844 (+ 621us) catalog_manager.cc:3725] Apply alter schema
0504 13:56:22.176711 (+ 5867us) catalog_manager.cc:3757] Acquired catalog manager lock
0504 13:56:22.176911 (+ 200us) catalog_manager.cc:3915] Updating metadata on disk
0504 13:56:22.495282 (+318371us) catalog_manager.cc:3946] Committing alterations to in-memory state
0504 13:56:22.496139 (+ 857us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:22.432857 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:22.433235 (+ 378us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:22.433257 (+ 22us) write_op.cc:435] Acquired schema lock
0504 13:56:22.433270 (+ 13us) tablet.cc:662] Decoding operations
0504 13:56:22.433508 (+ 238us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:22.433547 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:22.433569 (+ 22us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:22.433700 (+ 131us) tablet.cc:701] Row locks acquired
0504 13:56:22.433713 (+ 13us) write_op.cc:260] PREPARE: finished
0504 13:56:22.433807 (+ 94us) write_op.cc:270] Start()
0504 13:56:22.433885 (+ 78us) write_op.cc:276] Timestamp: P: 1777902982433765 usec, L: 0
0504 13:56:22.433901 (+ 16us) op_driver.cc:348] REPLICATION: starting
0504 13:56:22.477509 (+ 43608us) log.cc:844] Serialized 405 byte log entry
0504 13:56:22.478138 (+ 629us) spinlock_profiling.cc:243] Waited 43.1 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b801951 00007fc27b894208 00007fc27bd069f1 00007fc27bd061c8 00007fc27bd0ae61 00007fc27bd0ae19 00007fc27bd0ada9 00007fc27bd0ad71 00007fc27bd09f9d 00000000005b3bc4 00000000005b39f9
0504 13:56:22.491138 (+ 13000us) op_driver.cc:464] REPLICATION: finished
0504 13:56:22.491466 (+ 328us) write_op.cc:301] APPLY: starting
0504 13:56:22.491518 (+ 52us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:22.491616 (+ 98us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:22.491635 (+ 19us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:22.491762 (+ 127us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:22.491963 (+ 201us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:22.492011 (+ 48us) write_op.cc:312] APPLY: finished
0504 13:56:22.492179 (+ 168us) log.cc:844] Serialized 23 byte log entry
0504 13:56:22.492310 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:22.492369 (+ 59us) write_op.cc:454] Released schema lock
0504 13:56:22.492397 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:22.492598 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:22.492628 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":915847552,"child_traces":[["op",{"apply.queue_time_us":171,"apply.run_cpu_time_us":1103,"apply.run_wall_time_us":1103,"num_ops":1,"prepare.queue_time_us":254763,"prepare.run_cpu_time_us":2466,"prepare.run_wall_time_us":55221,"raft.queue_time_us":5280,"raft.run_cpu_time_us":748,"raft.run_wall_time_us":1626,"replication_time_us":57159,"spinlock_wait_cycles":99183488}]]}
I20260504 13:56:22.505780 10393 raft_consensus.cc:493] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:22.506206 10393 raft_consensus.cc:515] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.507815 9931 tablet_bootstrap.cc:654] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:22.508256 10393 leader_election.cc:290] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:22.510253 9932 tablet_bootstrap.cc:492] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:22.513697 9932 ts_tablet_manager.cc:1403] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.000s
I20260504 13:56:22.515323 9539 raft_consensus.cc:2468] T b13303d2a14c4187b82da9f25ce9753e P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:22.518170 9944 ts_tablet_manager.cc:1929] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:22.518501 9944 log.cc:1199] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/afad9329cf7745ba8168b2a93f998295
I20260504 13:56:22.518532 10391 ts_tablet_manager.cc:1916] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:22.519101 9944 ts_tablet_manager.cc:1950] T afad9329cf7745ba8168b2a93f998295 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:22.520812 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 140) took 1429 ms. Trace:
I20260504 13:56:22.521116 9944 rpcz_store.cc:276] 0504 13:56:21.091704 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.091854 (+ 150us) service_pool.cc:224] Handling call
0504 13:56:22.268742 (+1176888us) ts_tablet_manager.cc:1151] Deleting tablet afad9329cf7745ba8168b2a93f998295
0504 13:56:22.268780 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:22.446879 (+178099us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:22.518066 (+ 71187us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:22.520761 (+ 2695us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1176394}
I20260504 13:56:22.522192 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet afad9329cf7745ba8168b2a93f998295 (table default.Table_cdee25533fe54da0a74b421714b0d7c4 [id=81df524d85774544807ba1902be1aae1]) successfully deleted
I20260504 13:56:22.521939 9536 raft_consensus.cc:2468] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:22.523413 10393 raft_consensus.cc:493] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:22.428680 9511 rpcz_store.cc:276] 0504 13:56:21.268537 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.285710 (+ 17173us) service_pool.cc:224] Handling call
0504 13:56:21.332462 (+ 46752us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:21.332574 (+ 112us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:21.712657 (+380083us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:22.279506 (+566849us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:22.279880 (+ 374us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1673088}
I20260504 13:56:22.523792 10393 raft_consensus.cc:515] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.444806 10358 consensus_queue.cc:1048] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.081s
I20260504 13:56:22.526237 10393 leader_election.cc:290] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:22.527550 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5254b1061e2b424487f5d309baaee6b6" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:22.528637 9620 leader_election.cc:343] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.516485 9932 raft_consensus.cc:359] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.536027 9932 raft_consensus.cc:385] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:22.536604 9932 raft_consensus.cc:740] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
W20260504 13:56:22.536741 9335 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c12ffa802b840b1937f6f796083c478 has no consensus state
I20260504 13:56:22.537509 9932 consensus_queue.cc:260] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.540401 9335 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_9b8d10cfdddc499a86ba289a87dbe689" } modify_external_catalogs: true
I20260504 13:56:22.541152 9932 ts_tablet_manager.cc:1434] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.027s user 0.004s sys 0.000s
I20260504 13:56:22.540983 9335 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_9b8d10cfdddc499a86ba289a87dbe689" } modify_external_catalogs: true
I20260504 13:56:22.542377 9944 tablet_replica.cc:333] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:22.543040 9944 raft_consensus.cc:2243] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:22.543524 9944 raft_consensus.cc:2272] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:22.545114 9944 ts_tablet_manager.cc:1916] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:22.556952 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ed3ee9a13e84594905db47aa33040c2" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:22.560813 9620 leader_election.cc:343] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.568408 9932 tablet_bootstrap.cc:492] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:22.569464 9342 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35768 (request call id 216) took 1359 ms. Trace:
I20260504 13:56:22.569772 9342 rpcz_store.cc:276] 0504 13:56:21.210094 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.791855 (+581761us) service_pool.cc:224] Handling call
0504 13:56:22.193030 (+401175us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b0936 00007fc27b2bd524 00007fc27b2bd4b2 00007fc27b2bd3e1 00007fc27b2bd35c 00007fc27b2bc612 00007fc27958bbcc 00007fc27958b006
0504 13:56:22.195360 (+ 2330us) catalog_manager.cc:2101] Acquired catalog manager lock
0504 13:56:22.197001 (+ 1641us) catalog_manager.cc:2147] Created new table and tablet info
0504 13:56:22.568873 (+371872us) catalog_manager.cc:2207] Wrote table and tablets to system table
0504 13:56:22.569160 (+ 287us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0504 13:56:22.569412 (+ 252us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:22.531321 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:22.531784 (+ 463us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:22.531827 (+ 43us) write_op.cc:435] Acquired schema lock
0504 13:56:22.531845 (+ 18us) tablet.cc:662] Decoding operations
0504 13:56:22.555267 (+ 23422us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:22.555327 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:22.555362 (+ 35us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:22.555608 (+ 246us) tablet.cc:701] Row locks acquired
0504 13:56:22.555626 (+ 18us) write_op.cc:260] PREPARE: finished
0504 13:56:22.555760 (+ 134us) write_op.cc:270] Start()
0504 13:56:22.555857 (+ 97us) write_op.cc:276] Timestamp: P: 1777902982555701 usec, L: 0
0504 13:56:22.555878 (+ 21us) op_driver.cc:348] REPLICATION: starting
0504 13:56:22.559626 (+ 3748us) log.cc:844] Serialized 433 byte log entry
0504 13:56:22.566646 (+ 7020us) op_driver.cc:464] REPLICATION: finished
0504 13:56:22.567145 (+ 499us) write_op.cc:301] APPLY: starting
0504 13:56:22.567242 (+ 97us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:22.567369 (+ 127us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:22.567496 (+ 127us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:22.567972 (+ 476us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:22.568143 (+ 171us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0504 13:56:22.568200 (+ 57us) write_op.cc:312] APPLY: finished
0504 13:56:22.568406 (+ 206us) log.cc:844] Serialized 29 byte log entry
0504 13:56:22.568567 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:22.568650 (+ 83us) write_op.cc:454] Released schema lock
0504 13:56:22.568689 (+ 39us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":920355072,"child_traces":[["op",{"apply.queue_time_us":341,"apply.run_cpu_time_us":1725,"apply.run_wall_time_us":1731,"num_ops":2,"prepare.queue_time_us":332363,"raft.queue_time_us":92,"raft.run_cpu_time_us":737,"raft.run_wall_time_us":737,"replication_time_us":10652}]]}
I20260504 13:56:22.588822 9932 tablet_bootstrap.cc:654] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:22.601325 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be795f04dd54cf19869c937b91d0ddf" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:22.604597 9475 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.8.238.254:36145, user_credentials={real_user=slave}} blocked reactor thread for 79995.7us
I20260504 13:56:22.605880 9932 tablet_bootstrap.cc:492] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:22.606251 9932 ts_tablet_manager.cc:1403] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.001s
I20260504 13:56:22.612848 10414 consensus_queue.cc:1048] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:22.608608 9932 raft_consensus.cc:359] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.614621 9932 raft_consensus.cc:385] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:22.614889 9932 raft_consensus.cc:740] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:22.614622 10358 consensus_queue.cc:1059] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:22.615453 9944 ts_tablet_manager.cc:1929] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260504 13:56:22.615793 9944 log.cc:1199] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/2951cbd2df4f402eabeef323d121ce1b
I20260504 13:56:22.616484 9944 ts_tablet_manager.cc:1950] T 2951cbd2df4f402eabeef323d121ce1b P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:22.616568 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0be795f04dd54cf19869c937b91d0ddf" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:22.617334 9932 consensus_queue.cc:260] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.617682 10358 raft_consensus.cc:3055] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 13:56:22.618057 10358 raft_consensus.cc:740] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:22.620100 10358 consensus_queue.cc:260] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.621167 10358 raft_consensus.cc:3060] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Advancing to term 2
I20260504 13:56:22.625115 10393 raft_consensus.cc:493] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:22.625510 10393 raft_consensus.cc:515] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.627729 9618 leader_election.cc:304] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:22.629040 10466 raft_consensus.cc:2804] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:22.630242 10466 raft_consensus.cc:697] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:22.630421 9618 leader_election.cc:304] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:22.631021 10466 consensus_queue.cc:237] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.632611 10457 raft_consensus.cc:2804] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:22.635573 10393 leader_election.cc:290] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:22.637147 10013 tablet_bootstrap.cc:654] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:22.636960 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3c045379734134b49e94340679c11d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:22.639158 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ed3ee9a13e84594905db47aa33040c2" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:22.639429 9620 leader_election.cc:343] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.639854 10457 raft_consensus.cc:697] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:22.640215 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5254b1061e2b424487f5d309baaee6b6" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:22.640673 10457 consensus_queue.cc:237] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:22.658160 9339 master_service.cc:954] ReplaceTablet: received request to replace tablet c34877a81d504f46a2f680c5d5ddb4f4 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:22.659438 9932 ts_tablet_manager.cc:1434] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.053s user 0.007s sys 0.000s
I20260504 13:56:22.661019 9932 tablet_bootstrap.cc:492] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:22.661983 10369 consensus_queue.cc:1059] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:22.662704 10369 raft_consensus.cc:3055] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 13:56:22.662988 10369 raft_consensus.cc:740] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:22.663776 10369 consensus_queue.cc:260] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:22.664709 10369 raft_consensus.cc:3060] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Advancing to term 2
I20260504 13:56:22.664872 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 141) took 1537 ms. Trace:
I20260504 13:56:22.665160 9944 rpcz_store.cc:276] 0504 13:56:21.127603 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.127726 (+ 123us) service_pool.cc:224] Handling call
0504 13:56:22.542225 (+1414499us) ts_tablet_manager.cc:1151] Deleting tablet 2951cbd2df4f402eabeef323d121ce1b
0504 13:56:22.542262 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:22.577012 (+ 34750us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:22.614079 (+ 37067us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:22.664816 (+ 50737us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45254,"tablet-delete.queue_time_us":1414028}
I20260504 13:56:22.666684 9944 tablet_replica.cc:333] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:22.666510 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 2951cbd2df4f402eabeef323d121ce1b (table default.Table_d6ee06783219496b94ecc1a88d13b0da [id=46ac1f0baae0492984e87927e4905343]) successfully deleted
I20260504 13:56:22.667450 9944 raft_consensus.cc:2243] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:22.667900 9944 raft_consensus.cc:2272] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:22.669605 9944 ts_tablet_manager.cc:1916] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:22.669641 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3c045379734134b49e94340679c11d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:22.640698 10480 raft_consensus.cc:493] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:22.673264 10480 raft_consensus.cc:515] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:22.674649 9473 consensus_peers.cc:597] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70caac2e256d419dbc2e70cb5dfffe97. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:22.676159 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3112c3c3629412e9b76407c7cba8f92" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:22.677507 9620 leader_election.cc:343] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.678369 10480 leader_election.cc:290] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:22.689360 10382 raft_consensus.cc:493] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:22.689980 10382 raft_consensus.cc:515] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.693100 10382 leader_election.cc:290] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:22.695386 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd5f7d3f7c545d18fe19b64e4bc1de6" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:22.696039 9686 raft_consensus.cc:2468] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:22.697006 9768 leader_election.cc:304] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:22.675738 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34877a81d504f46a2f680c5d5ddb4f4" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:22.697937 10382 raft_consensus.cc:2804] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 13:56:22.698302 10382 raft_consensus.cc:493] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:22.698664 10382 raft_consensus.cc:3060] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Advancing to term 2
I20260504 13:56:22.702977 9536 raft_consensus.cc:2410] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260504 13:56:22.707259 9341 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_03e728dbf91f4fd19afe882c3b4d7bda" } new_table_name: "default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c" modify_external_catalogs: true
I20260504 13:56:22.709148 10382 raft_consensus.cc:515] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.714967 9932 tablet_bootstrap.cc:654] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:22.718006 10382 leader_election.cc:290] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:22.720757 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd5f7d3f7c545d18fe19b64e4bc1de6" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:22.722695 9686 raft_consensus.cc:3060] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Advancing to term 2
W20260504 13:56:22.727742 9473 leader_election.cc:343] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: c34630d55f4a4e19b33ddf4c13995a5b
W20260504 13:56:22.732309 9341 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7cb6927537704cc4b17617cbdaa21ac1 consensus state has no leader
I20260504 13:56:22.742286 9341 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_4b58a5b3e40c435b91e8fb70062833c1" } new_table_name: "default.Table_dee1335eff89438e9c68835a4c258031" modify_external_catalogs: true
I20260504 13:56:22.748049 9944 ts_tablet_manager.cc:1929] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260504 13:56:22.748486 9944 log.cc:1199] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/1cf2b59d13da40a2b2160ebdc0b2eb80
I20260504 13:56:22.749289 9944 ts_tablet_manager.cc:1950] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:22.755836 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 142) took 1090 ms. Trace:
W20260504 13:56:22.755985 9618 leader_election.cc:343] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.756361 9618 leader_election.cc:304] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:22.756150 9944 rpcz_store.cc:276] 0504 13:56:21.664878 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.665000 (+ 122us) service_pool.cc:224] Handling call
0504 13:56:22.666550 (+1001550us) ts_tablet_manager.cc:1151] Deleting tablet 1cf2b59d13da40a2b2160ebdc0b2eb80
0504 13:56:22.666591 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:22.732557 (+ 65966us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:22.747912 (+ 15355us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:22.754905 (+ 6993us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1000984}
I20260504 13:56:22.756990 10466 raft_consensus.cc:2749] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:22.757388 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 (table default.Table_0eae04160cf241df95cc53da6f4dfacf [id=3822050e2a9d4a9db8b0e36430fd0b7f]) successfully deleted
W20260504 13:56:22.758122 9618 leader_election.cc:343] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.758468 9618 leader_election.cc:304] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:22.759021 10466 raft_consensus.cc:2749] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:22.760753 9944 tablet_replica.cc:333] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
W20260504 13:56:22.760905 9618 leader_election.cc:343] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:22.761219 9618 leader_election.cc:304] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:22.761500 9944 raft_consensus.cc:2243] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260504 13:56:22.769235 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 0867df9ea115418193b6d9d3fc251923 because the tablet was not found. No further retry: Not found: Tablet not found: 0867df9ea115418193b6d9d3fc251923
W20260504 13:56:22.769966 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet b3793fe8f1ec43ad809403c4593f5ef6 because the tablet was not found. No further retry: Not found: Tablet not found: b3793fe8f1ec43ad809403c4593f5ef6
I20260504 13:56:22.770494 9473 leader_election.cc:304] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:22.771924 10414 raft_consensus.cc:2804] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:22.772559 10414 raft_consensus.cc:697] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:22.776825 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_df8874a20e834872a12300a4f5628273"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:22.778828 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df8874a20e834872a12300a4f5628273 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:22.773643 10414 consensus_queue.cc:237] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:22.799840 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_1230367a2b2e46f6bf1e4c8628f1eff1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:22.801764 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1230367a2b2e46f6bf1e4c8628f1eff1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:22.820071 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e4239942f34fb081a026a35740e465" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:22.831990 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1501455a0af348e38746d100987afdd0" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:22.834156 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd5f7d3f7c545d18fe19b64e4bc1de6" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:22.835583 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34877a81d504f46a2f680c5d5ddb4f4" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:22.836696 9536 raft_consensus.cc:2410] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260504 13:56:22.838117 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 67718.6us
W20260504 13:56:22.857242 9475 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.8.238.254:36145, user_credentials={real_user=slave}} blocked reactor thread for 105706us
I20260504 13:56:22.858835 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3112c3c3629412e9b76407c7cba8f92" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:22.864454 9472 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.Ping -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 65976.5us
I20260504 13:56:22.869351 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_003cbc6fdf0b4dddaba008af167dc621"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:22.876374 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_821e9f466c974a2fbf24c07f8c266e62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:22.876786 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_003cbc6fdf0b4dddaba008af167dc621 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:22.880524 10490 tablet.cc:1724] T b37a738f2902402ca76e4edebcdba474 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260504 13:56:22.881906 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_821e9f466c974a2fbf24c07f8c266e62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:22.886803 9686 raft_consensus.cc:2468] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 2.
I20260504 13:56:22.923640 9767 leader_election.cc:304] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:22.924799 9767 leader_election.cc:304] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:22.933300 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acd5f7d3f7c545d18fe19b64e4bc1de6" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:22.936208 9339 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_b32f665dddb043769e07076397f84a73" } new_table_name: "default.Table_72e3a254561b4cae927e6b6bcacd87b6" modify_external_catalogs: true
I20260504 13:56:22.944339 10494 raft_consensus.cc:2749] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:22.955582 9336 catalog_manager.cc:5671] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 2, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 2 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:22.956164 9518 tablet_service.cc:1511] Processing CreateTablet for tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 (DEFAULT_TABLE table=default.Table_0134ffe7086440ccbcadd076d8fe5d01 [id=693e9b6ed89b4ab7b9d5dc7402e17f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:22.956815 9336 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 0be795f04dd54cf19869c937b91d0ddf (table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]). Expected version 1 got 0
I20260504 13:56:22.957309 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41d41f8ba6dc4bb18116cb2b2dfabff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:22.957677 9336 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet b8e4239942f34fb081a026a35740e465 (table default.Table_11d8ab61a6044154bb599875a0bc9004 [id=73e12524eb9e416782bf9f7fb58c93bd]). Expected version 1 got 0
I20260504 13:56:22.958186 9336 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]). Expected version 1 got 0
I20260504 13:56:22.960310 9806 tablet_service.cc:1511] Processing CreateTablet for tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 (DEFAULT_TABLE table=default.Table_0134ffe7086440ccbcadd076d8fe5d01 [id=693e9b6ed89b4ab7b9d5dc7402e17f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:22.961372 9806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41d41f8ba6dc4bb18116cb2b2dfabff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:22.968735 9511 tablet_service.cc:1511] Processing CreateTablet for tablet 68e92cc3da9b4bcfadb6fa26245f1c99 (DEFAULT_TABLE table=default.Table_18f9a8030cea4cc0abedb81cd5accc59 [id=5c4f226dc33b428e92ca77959e075d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:22.969766 9511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68e92cc3da9b4bcfadb6fa26245f1c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:22.980911 9807 tablet_service.cc:1511] Processing CreateTablet for tablet 68e92cc3da9b4bcfadb6fa26245f1c99 (DEFAULT_TABLE table=default.Table_18f9a8030cea4cc0abedb81cd5accc59 [id=5c4f226dc33b428e92ca77959e075d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:22.982054 9807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68e92cc3da9b4bcfadb6fa26245f1c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:22.998234 9340 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 030b35c3409143108c34ab1f0ee1b96d
I20260504 13:56:22.999657 9340 catalog_manager.cc:6741] ReplaceTablet: tablet 030b35c3409143108c34ab1f0ee1b96d deleted and replaced by tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3
I20260504 13:56:23.002079 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet 030b35c3409143108c34ab1f0ee1b96d with delete_type TABLET_DATA_DELETED (replaced by tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:23.007344 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_1665799e268a496d9ffc5568fa34a763"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:23.008945 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1665799e268a496d9ffc5568fa34a763 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:23.009441 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 030b35c3409143108c34ab1f0ee1b96d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 030b35c3409143108c34ab1f0ee1b96d already in progress: creating tablet
W20260504 13:56:23.012302 9931 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.037s sys 0.154s
W20260504 13:56:23.012569 9931 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.503s user 0.037s sys 0.154s
I20260504 13:56:23.016462 9931 tablet_bootstrap.cc:492] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:23.018759 9931 ts_tablet_manager.cc:1403] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.522s user 0.040s sys 0.157s
I20260504 13:56:23.018901 10494 raft_consensus.cc:493] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:23.019342 10494 raft_consensus.cc:515] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.021178 10494 leader_election.cc:290] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:23.010356 9474 consensus_peers.cc:597] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07a428049fdb4a68aae729400e2d9674. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:23.025288 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_89c38f9ba46e48fab397a7f3d5d1b805"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:23.026705 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89c38f9ba46e48fab397a7f3d5d1b805 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:23.030261 9807 tablet_service.cc:1511] Processing CreateTablet for tablet bae90898739b4303af0608e39e8c0367 (DEFAULT_TABLE table=default.Table_073a6dae9da444bea1f96b2f33fe1a85 [id=39c692c399dd43ca963357b6818871cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.031286 9807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae90898739b4303af0608e39e8c0367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.032646 10486 tablet.cc:1724] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260504 13:56:23.035567 9474 leader_election.cc:343] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:23.036015 9474 leader_election.cc:304] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:23.038089 10382 raft_consensus.cc:2749] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:23.036795 9508 tablet_service.cc:1558] Processing DeleteTablet for tablet 030b35c3409143108c34ab1f0ee1b96d with delete_type TABLET_DATA_DELETED (replaced by tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:23.051486 10013 tablet_bootstrap.cc:492] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:23.061098 10013 ts_tablet_manager.cc:1403] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.807s user 0.009s sys 0.003s
I20260504 13:56:23.069159 9931 raft_consensus.cc:359] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.049274 10489 tablet.cc:1724] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260504 13:56:23.066596 9474 raft_consensus.cc:2686] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260504 13:56:23.079186 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 43630.4us
I20260504 13:56:22.957620 9656 tablet_service.cc:1511] Processing CreateTablet for tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 (DEFAULT_TABLE table=default.Table_0134ffe7086440ccbcadd076d8fe5d01 [id=693e9b6ed89b4ab7b9d5dc7402e17f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.086083 9656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41d41f8ba6dc4bb18116cb2b2dfabff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.088222 9508 tablet_service.cc:1511] Processing CreateTablet for tablet bae90898739b4303af0608e39e8c0367 (DEFAULT_TABLE table=default.Table_073a6dae9da444bea1f96b2f33fe1a85 [id=39c692c399dd43ca963357b6818871cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.089246 9508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae90898739b4303af0608e39e8c0367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:22.962642 9659 tablet_service.cc:1511] Processing CreateTablet for tablet 68e92cc3da9b4bcfadb6fa26245f1c99 (DEFAULT_TABLE table=default.Table_18f9a8030cea4cc0abedb81cd5accc59 [id=5c4f226dc33b428e92ca77959e075d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.091777 9659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68e92cc3da9b4bcfadb6fa26245f1c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.000711 9657 tablet_service.cc:1558] Processing DeleteTablet for tablet 030b35c3409143108c34ab1f0ee1b96d with delete_type TABLET_DATA_DELETED (replaced by tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:23.028501 9663 tablet_service.cc:1511] Processing CreateTablet for tablet bae90898739b4303af0608e39e8c0367 (DEFAULT_TABLE table=default.Table_073a6dae9da444bea1f96b2f33fe1a85 [id=39c692c399dd43ca963357b6818871cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.072846 9932 tablet_bootstrap.cc:492] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:23.094076 9932 ts_tablet_manager.cc:1403] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.433s user 0.068s sys 0.111s
I20260504 13:56:23.096599 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae90898739b4303af0608e39e8c0367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.118220 10462 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:52642 (request call id 135) took 1852 ms. Trace:
I20260504 13:56:23.120824 10500 tablet.cc:1724] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:23.122810 9931 raft_consensus.cc:385] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:23.123407 9931 raft_consensus.cc:740] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
W20260504 13:56:23.131628 9473 consensus_peers.cc:597] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07a428049fdb4a68aae729400e2d9674. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:23.131824 10429 raft_consensus.cc:493] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:23.132282 10429 raft_consensus.cc:515] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.134658 10429 leader_election.cc:290] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:23.138495 10391 ts_tablet_manager.cc:1929] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260504 13:56:23.139706 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 59076.2us
I20260504 13:56:23.143013 10429 consensus_queue.cc:1048] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:23.148017 10493 log.cc:927] Time spent T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20260504 13:56:23.150120 10013 raft_consensus.cc:359] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:23.150754 10013 raft_consensus.cc:385] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:23.118600 10462 rpcz_store.cc:276] 0504 13:56:21.266191 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.290447 (+ 24256us) service_pool.cc:224] Handling call
0504 13:56:23.118140 (+1827693us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:21.468082 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0504 13:56:21.469720 (+ 1638us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0504 13:56:21.469773 (+ 53us) alter_schema_op.cc:66] Acquired schema lock
0504 13:56:21.472003 (+ 2230us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0504 13:56:21.472189 (+ 186us) alter_schema_op.cc:128] START. Timestamp: 1777902981472037
0504 13:56:21.472244 (+ 55us) op_driver.cc:348] REPLICATION: starting
0504 13:56:21.472984 (+ 740us) log.cc:844] Serialized 173 byte log entry
0504 13:56:22.200405 (+727421us) op_driver.cc:464] REPLICATION: finished
0504 13:56:22.297336 (+ 96931us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0504 13:56:22.533858 (+236522us) log.cc:844] Serialized 5 byte log entry
0504 13:56:23.080688 (+546830us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:23.117880 (+ 37192us) log.cc:844] Serialized 14 byte log entry
0504 13:56:23.117982 (+ 102us) alter_schema_op.cc:73] Released schema lock
0504 13:56:23.118016 (+ 34us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"spinlock_wait_cycles":538368,"child_traces":[["op",{"apply.queue_time_us":96749,"mutex_wait_us":7800,"prepare.queue_time_us":104288,"prepare.run_cpu_time_us":14183,"prepare.run_wall_time_us":405392,"raft.queue_time_us":54631,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":10,"replication_time_us":728034,"thread_start_us":416675,"threads_started":2,"wal-append.queue_time_us":278438}]]}
I20260504 13:56:23.146234 10391 log.cc:1199] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/c34630d55f4a4e19b33ddf4c13995a5b
I20260504 13:56:23.164465 10391 ts_tablet_manager.cc:1950] T c34630d55f4a4e19b33ddf4c13995a5b P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:23.165452 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 127) took 2673 ms. Trace:
W20260504 13:56:23.165882 9474 consensus_peers.cc:597] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65052a5c28c147a385144317e21eea41. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:23.166580 9474 consensus_peers.cc:597] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c43ff1d759a40d68344f25f066416ac. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:23.166676 9807 tablet_service.cc:1511] Processing CreateTablet for tablet 1c12ffa802b840b1937f6f796083c478 (DEFAULT_TABLE table=default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.169703 9807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c12ffa802b840b1937f6f796083c478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.169528 9931 consensus_queue.cc:260] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.165839 10391 rpcz_store.cc:276] 0504 13:56:20.491621 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:20.500576 (+ 8955us) service_pool.cc:224] Handling call
0504 13:56:22.195407 (+1694831us) ts_tablet_manager.cc:1151] Deleting tablet c34630d55f4a4e19b33ddf4c13995a5b
0504 13:56:22.195461 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:22.291475 (+ 96014us) spinlock_profiling.cc:243] Waited 69.6 ms on lock 0x7b5400111f54. stack: 00007f591ed51c83 00007f591ed51b80 00007f591e8c2574 000000000050e094 000000000050e029 00007f5925da9951 00007f5925e48742 00007f592626b7bc 00007f5926ec3cca 00007f5926ef4990 00007f5926edaefd 00007f5926edaea9 00007f5926edae39 00007f5926edae01 00007f5926eda02d 00000000005b3bc4
0504 13:56:22.830894 (+539419us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:23.138378 (+307484us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:23.165386 (+ 27008us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82028,"spinlock_wait_cycles":160056064,"tablet-delete.queue_time_us":1693806}
I20260504 13:56:23.169672 9520 tablet_service.cc:1511] Processing CreateTablet for tablet 1c12ffa802b840b1937f6f796083c478 (DEFAULT_TABLE table=default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.180073 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c12ffa802b840b1937f6f796083c478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.181164 9344 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_8ccb9fc27b374611b20008514d3ad483"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:23.182660 9344 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ccb9fc27b374611b20008514d3ad483 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:23.185201 10462 tablet.cc:1724] T b8e4239942f34fb081a026a35740e465 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:23.185845 9833 raft_consensus.cc:1240] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [term 2 FOLLOWER]: Rejecting Update request from peer 7b79fe7dd8934dd996393a2d96eabf00 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260504 13:56:23.186338 10391 tablet_replica.cc:333] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:23.186800 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afad9329cf7745ba8168b2a93f998295" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:23.151087 10013 raft_consensus.cc:740] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:23.191211 10490 tablet.cc:1724] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:23.191680 10391 raft_consensus.cc:2243] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:23.191882 10013 consensus_queue.cc:260] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:23.195521 9931 ts_tablet_manager.cc:1434] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.177s user 0.006s sys 0.000s
I20260504 13:56:23.196784 9931 tablet_bootstrap.cc:492] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:23.200996 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet c34630d55f4a4e19b33ddf4c13995a5b (table default.Table_434bdc9867ad475a9719ba0c3760ef71 [id=db5cbb438d8f4da981e70b12713134f1]) successfully deleted
I20260504 13:56:23.201923 10490 tablet.cc:1724] T b8e4239942f34fb081a026a35740e465 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:23.192297 10391 raft_consensus.cc:2272] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:23.205936 10391 ts_tablet_manager.cc:1916] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:23.214381 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b13303d2a14c4187b82da9f25ce9753e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:23.215102 9540 raft_consensus.cc:2393] T b13303d2a14c4187b82da9f25ce9753e P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 5beceb6317444247b31b44a6581707c6 in this term.
I20260504 13:56:23.217993 9540 raft_consensus.cc:1275] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: Refusing update from remote peer d4051d2bf0bf410eb8acd107560df888: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260504 13:56:23.218402 9540 pending_rounds.cc:85] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00: Aborting all ops after (but not including) 0
I20260504 13:56:23.218709 9540 pending_rounds.cc:107] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260504 13:56:23.219033 9540 raft_consensus.cc:2889] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 13:56:23.224556 10489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:52642 (request call id 149) took 1256 ms. Trace:
I20260504 13:56:23.194011 10013 ts_tablet_manager.cc:1434] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.133s user 0.006s sys 0.000s
I20260504 13:56:23.228544 9540 raft_consensus.cc:1275] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: Refusing update from remote peer d4051d2bf0bf410eb8acd107560df888: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260504 13:56:23.228933 9540 pending_rounds.cc:85] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Aborting all ops after (but not including) 0
I20260504 13:56:23.229195 9540 pending_rounds.cc:107] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260504 13:56:23.229460 9540 raft_consensus.cc:2889] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 13:56:23.233783 10431 consensus_queue.cc:1059] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:23.234652 10414 raft_consensus.cc:3055] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 13:56:23.234993 10414 raft_consensus.cc:740] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:23.227053 10013 tablet_bootstrap.cc:492] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:23.235827 10414 consensus_queue.cc:260] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.239864 10414 raft_consensus.cc:3060] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Advancing to term 2
I20260504 13:56:23.241544 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_69c9551ae2cf4ef399e40017a33bad73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:23.242838 9337 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 930078a10afa4dd987ad021cf9609514
W20260504 13:56:23.242938 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69c9551ae2cf4ef399e40017a33bad73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:23.245355 9337 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35768 (request call id 217) took 1915 ms. Trace:
W20260504 13:56:23.246709 9474 leader_election.cc:343] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:23.258426 9338 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet b8e4239942f34fb081a026a35740e465 (table default.Table_11d8ab61a6044154bb599875a0bc9004 [id=73e12524eb9e416782bf9f7fb58c93bd]). Expected version 1 got 0
I20260504 13:56:23.224944 10489 rpcz_store.cc:276] 0504 13:56:21.968571 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.975023 (+ 6452us) service_pool.cc:224] Handling call
0504 13:56:23.224502 (+1249479us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:22.120302 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0504 13:56:22.120640 (+ 338us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0504 13:56:22.120670 (+ 30us) alter_schema_op.cc:66] Acquired schema lock
0504 13:56:22.120736 (+ 66us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0504 13:56:22.120844 (+ 108us) alter_schema_op.cc:128] START. Timestamp: 1777902982119491
0504 13:56:22.120863 (+ 19us) op_driver.cc:348] REPLICATION: starting
0504 13:56:22.121336 (+ 473us) log.cc:844] Serialized 173 byte log entry
0504 13:56:22.839744 (+718408us) op_driver.cc:464] REPLICATION: finished
0504 13:56:23.037356 (+197612us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0504 13:56:23.080883 (+ 43527us) log.cc:844] Serialized 5 byte log entry
0504 13:56:23.162391 (+ 81508us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:23.223792 (+ 61401us) log.cc:844] Serialized 14 byte log entry
0504 13:56:23.224365 (+ 573us) alter_schema_op.cc:73] Released schema lock
0504 13:56:23.224393 (+ 28us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":197375,"mutex_wait_us":69323,"prepare.queue_time_us":111045,"prepare.run_cpu_time_us":1967,"prepare.run_wall_time_us":229186,"raft.queue_time_us":71358,"raft.run_cpu_time_us":654,"raft.run_wall_time_us":652,"replication_time_us":718734,"thread_start_us":464567,"threads_started":3,"wal-append.queue_time_us":36262}]]}
I20260504 13:56:23.245625 9337 rpcz_store.cc:276] 0504 13:56:21.329708 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.796744 (+467036us) service_pool.cc:224] Handling call
0504 13:56:22.147109 (+350365us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b0c46 00007fc27b2c2024 00007fc27b2c1fb2 00007fc27b2c1ee1 00007fc27b2c1e5c 00007fc27b2c1112 00007fc27958bbcc 00007fc27958b006
0504 13:56:22.152221 (+ 5112us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:22.152700 (+ 479us) catalog_manager.cc:2815] Modifying in-memory table state
0504 13:56:22.152819 (+ 119us) catalog_manager.cc:2824] Locking tablets
0504 13:56:22.949282 (+796463us) catalog_manager.cc:2838] Removing table and tablets from system table
0504 13:56:23.242508 (+293226us) catalog_manager.cc:2856] Removing table from by-name map
0504 13:56:23.242678 (+ 170us) catalog_manager.cc:2872] Committing in-memory state
0504 13:56:23.242763 (+ 85us) catalog_manager.cc:2876] Aborting table tasks
0504 13:56:23.245300 (+ 2537us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:22.969734 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:22.970155 (+ 421us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:22.970186 (+ 31us) write_op.cc:435] Acquired schema lock
0504 13:56:22.970206 (+ 20us) tablet.cc:662] Decoding operations
0504 13:56:23.083347 (+113141us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:23.083414 (+ 67us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:23.083449 (+ 35us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:23.083804 (+ 355us) tablet.cc:701] Row locks acquired
0504 13:56:23.083824 (+ 20us) write_op.cc:260] PREPARE: finished
0504 13:56:23.083962 (+ 138us) write_op.cc:270] Start()
0504 13:56:23.084077 (+ 115us) write_op.cc:276] Timestamp: P: 1777902983083894 usec, L: 0
0504 13:56:23.084100 (+ 23us) op_driver.cc:348] REPLICATION: starting
0504 13:56:23.084726 (+ 626us) log.cc:844] Serialized 762 byte log entry
0504 13:56:23.236468 (+151742us) op_driver.cc:464] REPLICATION: finished
0504 13:56:23.237501 (+ 1033us) write_op.cc:301] APPLY: starting
0504 13:56:23.237560 (+ 59us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:23.237653 (+ 93us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:23.237733 (+ 80us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:23.237997 (+ 264us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:23.238114 (+ 117us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:23.238153 (+ 39us) write_op.cc:312] APPLY: finished
0504 13:56:23.238305 (+ 152us) log.cc:844] Serialized 29 byte log entry
0504 13:56:23.238431 (+ 126us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:23.238493 (+ 62us) write_op.cc:454] Released schema lock
0504 13:56:23.238519 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:23.238715 (+ 196us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:23.238741 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":803990400,"child_traces":[["op",{"apply.queue_time_us":898,"apply.run_cpu_time_us":1181,"apply.run_wall_time_us":1181,"num_ops":2,"prepare.queue_time_us":14548,"prepare.run_cpu_time_us":2671,"prepare.run_wall_time_us":238005,"raft.queue_time_us":1970,"raft.run_cpu_time_us":721,"raft.run_wall_time_us":721,"replication_time_us":152264}]]}
I20260504 13:56:23.261601 9338 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet b37a738f2902402ca76e4edebcdba474 (table default.Table_91cc6c38c9ff43dca25814664068b1c6 [id=da5c427e75e74ea1ae3a5961ed27e0de]). Expected version 1 got 0
I20260504 13:56:23.263137 9338 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:23.264001 9338 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 0be795f04dd54cf19869c937b91d0ddf (table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]). Expected version 1 got 0
I20260504 13:56:23.265136 9338 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 604f1ba5f71041a7877daff8972a994a (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]). Expected version 2 got 0
I20260504 13:56:23.266942 10462 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:52642 (request call id 150) took 1298 ms. Trace:
I20260504 13:56:23.267304 10462 rpcz_store.cc:276] 0504 13:56:21.968828 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.968921 (+ 93us) service_pool.cc:224] Handling call
0504 13:56:23.266888 (+1297967us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:22.062117 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0504 13:56:22.078762 (+ 16645us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0504 13:56:22.078792 (+ 30us) alter_schema_op.cc:66] Acquired schema lock
0504 13:56:22.078850 (+ 58us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0504 13:56:22.078969 (+ 119us) alter_schema_op.cc:128] START. Timestamp: 1777902982077622
0504 13:56:22.078989 (+ 20us) op_driver.cc:348] REPLICATION: starting
0504 13:56:22.096214 (+ 17225us) log.cc:844] Serialized 173 byte log entry
0504 13:56:23.181707 (+1085493us) op_driver.cc:464] REPLICATION: finished
0504 13:56:23.185100 (+ 3393us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0504 13:56:23.186369 (+ 1269us) log.cc:844] Serialized 5 byte log entry
0504 13:56:23.253703 (+ 67334us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:23.265537 (+ 11834us) log.cc:844] Serialized 14 byte log entry
0504 13:56:23.266737 (+ 1200us) alter_schema_op.cc:73] Released schema lock
0504 13:56:23.266771 (+ 34us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3207,"prepare.queue_time_us":904,"prepare.run_cpu_time_us":1796,"prepare.run_wall_time_us":60768,"raft.queue_time_us":88,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":14,"replication_time_us":1102611,"thread_start_us":47058,"threads_started":2,"wal-append.queue_time_us":31824,"wal-append.run_cpu_time_us":1507,"wal-append.run_wall_time_us":1004143}]]}
I20260504 13:56:23.270727 10382 consensus_queue.cc:1048] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:23.274564 10013 tablet_bootstrap.cc:654] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:23.281494 9658 tablet_service.cc:1558] Processing DeleteTablet for tablet 930078a10afa4dd987ad021cf9609514 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:23.281626 9473 leader_election.cc:343] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: afad9329cf7745ba8168b2a93f998295
I20260504 13:56:23.282083 9473 leader_election.cc:304] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:23.282850 10414 raft_consensus.cc:2749] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:23.286082 9657 tablet_service.cc:1511] Processing CreateTablet for tablet 1c12ffa802b840b1937f6f796083c478 (DEFAULT_TABLE table=default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.287361 9657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c12ffa802b840b1937f6f796083c478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.287258 9511 tablet_service.cc:1558] Processing DeleteTablet for tablet 930078a10afa4dd987ad021cf9609514 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:23.287850 9338 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]). Expected version 1 got 0
I20260504 13:56:23.288403 9338 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:23.289290 9338 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet b13303d2a14c4187b82da9f25ce9753e (table default.Table_b32f665dddb043769e07076397f84a73 [id=f227576f105c4b1fbd459b3638ce684b]). Expected version 1 got 0
I20260504 13:56:23.293784 9685 raft_consensus.cc:1240] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [term 2 FOLLOWER]: Rejecting Update request from peer 7b79fe7dd8934dd996393a2d96eabf00 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260504 13:56:23.301086 9806 tablet_service.cc:1511] Processing CreateTablet for tablet fac3e36cfa4848a1ab015f89c56c2740 (DEFAULT_TABLE table=default.Table_8cae213b77994b9fba63b8a7c5f27249 [id=df86bccb4ca8414c90f9fd8753724d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.302556 9806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fac3e36cfa4848a1ab015f89c56c2740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.291288 10382 consensus_queue.cc:1048] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260504 13:56:23.304804 10466 raft_consensus.cc:2749] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:23.305253 9944 raft_consensus.cc:2272] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:23.305649 9618 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 544755us
I20260504 13:56:23.307008 9944 ts_tablet_manager.cc:1916] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:23.309773 9768 leader_election.cc:304] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:23.313328 10382 raft_consensus.cc:2804] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [term 2 FOLLOWER]: Leader election won for term 2
I20260504 13:56:23.313756 10382 raft_consensus.cc:697] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [term 2 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:23.314570 10382 consensus_queue.cc:237] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.326479 10505 tablet.cc:1724] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:23.323753 9807 tablet_service.cc:1558] Processing DeleteTablet for tablet 930078a10afa4dd987ad021cf9609514 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:56380
W20260504 13:56:23.338312 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 930078a10afa4dd987ad021cf9609514 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 930078a10afa4dd987ad021cf9609514 already in progress: creating tablet
I20260504 13:56:23.338925 9931 tablet_bootstrap.cc:654] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:23.340637 9932 raft_consensus.cc:359] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.341290 9932 raft_consensus.cc:385] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:23.341535 9932 raft_consensus.cc:740] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:23.345249 9932 consensus_queue.cc:260] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.347991 9658 tablet_service.cc:1511] Processing CreateTablet for tablet fac3e36cfa4848a1ab015f89c56c2740 (DEFAULT_TABLE table=default.Table_8cae213b77994b9fba63b8a7c5f27249 [id=df86bccb4ca8414c90f9fd8753724d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.339380 10457 raft_consensus.cc:493] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:23.350632 10457 raft_consensus.cc:515] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.352552 10457 leader_election.cc:290] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:23.353775 9511 tablet_service.cc:1511] Processing CreateTablet for tablet fac3e36cfa4848a1ab015f89c56c2740 (DEFAULT_TABLE table=default.Table_8cae213b77994b9fba63b8a7c5f27249 [id=df86bccb4ca8414c90f9fd8753724d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.396343 10500 tablet.cc:1724] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260504 13:56:23.399264 9768 consensus_peers.cc:597] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:23.399545 9683 raft_consensus.cc:1275] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6 [term 2 FOLLOWER]: Refusing update from remote peer d4051d2bf0bf410eb8acd107560df888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260504 13:56:23.411427 9658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fac3e36cfa4848a1ab015f89c56c2740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.412420 10497 consensus_queue.cc:1048] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:23.421478 9931 tablet_bootstrap.cc:492] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:23.421940 9931 ts_tablet_manager.cc:1403] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.225s user 0.010s sys 0.002s
I20260504 13:56:23.414844 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b13303d2a14c4187b82da9f25ce9753e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:23.428613 9768 leader_election.cc:304] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:23.429636 10497 raft_consensus.cc:3060] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:23.431607 10013 tablet_bootstrap.cc:492] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:23.432049 10013 ts_tablet_manager.cc:1403] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.205s user 0.005s sys 0.007s
I20260504 13:56:23.437878 10497 raft_consensus.cc:2749] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:23.453939 10013 raft_consensus.cc:359] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.454668 10013 raft_consensus.cc:385] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:23.455621 10499 tablet.cc:1724] T b8e4239942f34fb081a026a35740e465 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:23.462186 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_7dbd11258f34498e82a86f7e3c72f470"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:23.463801 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7dbd11258f34498e82a86f7e3c72f470 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:23.469578 9931 raft_consensus.cc:359] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.470326 9931 raft_consensus.cc:385] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:23.470583 9931 raft_consensus.cc:740] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:23.471442 9931 consensus_queue.cc:260] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.474771 9342 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_a8bc83249ad949c49e759a07818e2c02" } modify_external_catalogs: true
I20260504 13:56:23.475445 9342 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_a8bc83249ad949c49e759a07818e2c02" } modify_external_catalogs: true
I20260504 13:56:23.481961 10391 ts_tablet_manager.cc:1929] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260504 13:56:23.484912 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb6927537704cc4b17617cbdaa21ac1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:23.454958 10013 raft_consensus.cc:740] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
W20260504 13:56:23.503978 9620 leader_election.cc:343] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:23.501089 10013 consensus_queue.cc:260] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.514966 10013 ts_tablet_manager.cc:1434] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.064s user 0.002s sys 0.004s
I20260504 13:56:23.511580 9931 ts_tablet_manager.cc:1434] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.089s user 0.007s sys 0.001s
I20260504 13:56:23.482434 10391 log.cc:1199] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/2951cbd2df4f402eabeef323d121ce1b
I20260504 13:56:23.518517 9830 raft_consensus.cc:3060] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:23.519227 10391 ts_tablet_manager.cc:1950] T 2951cbd2df4f402eabeef323d121ce1b P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:23.519919 9931 tablet_bootstrap.cc:492] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:23.523766 10486 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:52642 (request call id 151) took 1554 ms. Trace:
I20260504 13:56:23.528952 10486 rpcz_store.cc:276] 0504 13:56:21.969085 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.969178 (+ 93us) service_pool.cc:224] Handling call
0504 13:56:23.523696 (+1554518us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:22.009319 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0504 13:56:22.013149 (+ 3830us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0504 13:56:22.013180 (+ 31us) alter_schema_op.cc:66] Acquired schema lock
0504 13:56:22.014018 (+ 838us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0504 13:56:22.014148 (+ 130us) alter_schema_op.cc:128] START. Timestamp: 1777902982012797
0504 13:56:22.014168 (+ 20us) op_driver.cc:348] REPLICATION: starting
0504 13:56:22.015004 (+ 836us) log.cc:844] Serialized 173 byte log entry
0504 13:56:22.705495 (+690491us) op_driver.cc:464] REPLICATION: finished
0504 13:56:23.032557 (+327062us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0504 13:56:23.034127 (+ 1570us) log.cc:844] Serialized 5 byte log entry
0504 13:56:23.162304 (+128177us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:23.522795 (+360491us) log.cc:844] Serialized 14 byte log entry
0504 13:56:23.523523 (+ 728us) alter_schema_op.cc:73] Released schema lock
0504 13:56:23.523564 (+ 41us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":299705,"prepare.queue_time_us":594,"prepare.run_cpu_time_us":11655,"prepare.run_wall_time_us":237549,"replication_time_us":691215,"thread_start_us":508302,"threads_started":2,"wal-append.queue_time_us":148356}]]}
I20260504 13:56:23.537017 9932 ts_tablet_manager.cc:1434] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.443s user 0.010s sys 0.003s
I20260504 13:56:23.538007 9932 tablet_bootstrap.cc:492] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:23.540099 9618 leader_election.cc:343] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:23.540580 9618 leader_election.cc:304] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:23.554739 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_2f0fc59ee89c402cb8a0fc23341636e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:23.556352 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f0fc59ee89c402cb8a0fc23341636e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:23.568162 9511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fac3e36cfa4848a1ab015f89c56c2740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.579591 10013 tablet_bootstrap.cc:492] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:23.579861 9520 tablet_service.cc:1511] Processing CreateTablet for tablet c6ebc96408d64c2da7dc6c4fc158995c (DEFAULT_TABLE table=default.Table_5ff6dab0e47d41e681533fca02a36153 [id=230a8885dd2b4995815b650d3d9f9f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.580822 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6ebc96408d64c2da7dc6c4fc158995c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.580680 9806 tablet_service.cc:1511] Processing CreateTablet for tablet c6ebc96408d64c2da7dc6c4fc158995c (DEFAULT_TABLE table=default.Table_5ff6dab0e47d41e681533fca02a36153 [id=230a8885dd2b4995815b650d3d9f9f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.582450 9807 tablet_service.cc:1511] Processing CreateTablet for tablet 4fae9274bb8240798f85b79504a227b3 (DEFAULT_TABLE table=default.Table_e116f9abbb23419d9a3365f7d0e17db9 [id=bee8acaa102249578514a3d4e5f876d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.583452 9807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fae9274bb8240798f85b79504a227b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.591579 9806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6ebc96408d64c2da7dc6c4fc158995c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.591745 9663 tablet_service.cc:1511] Processing CreateTablet for tablet c6ebc96408d64c2da7dc6c4fc158995c (DEFAULT_TABLE table=default.Table_5ff6dab0e47d41e681533fca02a36153 [id=230a8885dd2b4995815b650d3d9f9f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.592760 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6ebc96408d64c2da7dc6c4fc158995c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.601750 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 147) took 2327 ms. Trace:
I20260504 13:56:23.602272 10391 rpcz_store.cc:276] 0504 13:56:21.274279 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.281061 (+ 6782us) service_pool.cc:224] Handling call
0504 13:56:23.186185 (+1905124us) ts_tablet_manager.cc:1151] Deleting tablet 2951cbd2df4f402eabeef323d121ce1b
0504 13:56:23.186231 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:23.342374 (+156143us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:23.481795 (+139421us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:23.601681 (+119886us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1903864}
I20260504 13:56:23.606055 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 2951cbd2df4f402eabeef323d121ce1b (table default.Table_d6ee06783219496b94ecc1a88d13b0da [id=46ac1f0baae0492984e87927e4905343]) successfully deleted
I20260504 13:56:23.606688 10513 raft_consensus.cc:493] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:23.607445 10513 raft_consensus.cc:515] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.609691 10391 tablet_replica.cc:333] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:23.611294 10513 leader_election.cc:290] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:23.612186 10513 raft_consensus.cc:2749] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:23.619606 10013 tablet_bootstrap.cc:654] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:23.619854 9618 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 78148.4us
I20260504 13:56:23.622018 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb6927537704cc4b17617cbdaa21ac1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:23.625237 10510 tablet.cc:1724] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260504 13:56:23.635994 9328 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c47b6e3b4d0c45c28384f7aae6b749de consensus state has no leader
I20260504 13:56:23.658473 9932 tablet_bootstrap.cc:654] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:23.664624 9343 catalog_manager.cc:5671] T 03fe4ce4f52f48b7b7a804c269e0fd4e P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:23.666461 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:23.667796 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 0be795f04dd54cf19869c937b91d0ddf (table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]). Expected version 1 got 0
I20260504 13:56:23.670358 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet b37a738f2902402ca76e4edebcdba474 (table default.Table_91cc6c38c9ff43dca25814664068b1c6 [id=da5c427e75e74ea1ae3a5961ed27e0de]). Expected version 1 got 0
I20260504 13:56:23.673588 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]). Expected version 1 got 0
I20260504 13:56:23.675464 10391 raft_consensus.cc:2243] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:23.676451 10391 pending_rounds.cc:70] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00: Trying to abort 1 pending ops.
I20260504 13:56:23.676733 10391 pending_rounds.cc:77] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290606539866112 op_type: NO_OP noop_request { }
I20260504 13:56:23.676427 9343 catalog_manager.cc:5671] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:23.677157 10391 raft_consensus.cc:2889] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:23.677544 10391 raft_consensus.cc:2272] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:23.679826 10391 ts_tablet_manager.cc:1916] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:23.685832 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet b8e4239942f34fb081a026a35740e465 (table default.Table_11d8ab61a6044154bb599875a0bc9004 [id=73e12524eb9e416782bf9f7fb58c93bd]). Expected version 1 got 0
I20260504 13:56:23.686422 9342 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 8e4e583afc804212bece2f727411cc14
I20260504 13:56:23.686488 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet b13303d2a14c4187b82da9f25ce9753e (table default.Table_b32f665dddb043769e07076397f84a73 [id=f227576f105c4b1fbd459b3638ce684b]). Expected version 1 got 0
I20260504 13:56:23.696471 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet 8e4e583afc804212bece2f727411cc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:23 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:23.696035 10522 raft_consensus.cc:493] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:23.697386 10522 raft_consensus.cc:515] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.698562 9931 tablet_bootstrap.cc:654] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:23.699435 10522 leader_election.cc:290] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:23.691598 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "930078a10afa4dd987ad021cf9609514" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:23.694425 9511 tablet_service.cc:1558] Processing DeleteTablet for tablet 8e4e583afc804212bece2f727411cc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:23 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:23.695358 10486 tablet.cc:1724] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:23.703828 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "930078a10afa4dd987ad021cf9609514" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:23.706550 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 8e4e583afc804212bece2f727411cc14 because the tablet was not found. No further retry: Not found: Tablet not found: 8e4e583afc804212bece2f727411cc14
I20260504 13:56:23.707867 10414 consensus_queue.cc:1048] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:23.717401 9511 tablet_service.cc:1511] Processing CreateTablet for tablet 4fae9274bb8240798f85b79504a227b3 (DEFAULT_TABLE table=default.Table_e116f9abbb23419d9a3365f7d0e17db9 [id=bee8acaa102249578514a3d4e5f876d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.718374 9511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fae9274bb8240798f85b79504a227b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.721835 10510 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:40388 (request call id 149) took 1415 ms. Trace:
I20260504 13:56:23.722247 10510 rpcz_store.cc:276] 0504 13:56:22.306242 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:22.306474 (+ 232us) service_pool.cc:224] Handling call
0504 13:56:23.721779 (+1415305us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:22.307449 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0504 13:56:22.307752 (+ 303us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0504 13:56:22.307779 (+ 27us) alter_schema_op.cc:66] Acquired schema lock
0504 13:56:22.307818 (+ 39us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0504 13:56:22.307924 (+ 106us) alter_schema_op.cc:128] START. Timestamp: 1777902982307853
0504 13:56:22.307953 (+ 29us) op_driver.cc:348] REPLICATION: starting
0504 13:56:22.308418 (+ 465us) log.cc:844] Serialized 173 byte log entry
0504 13:56:23.624685 (+1316267us) op_driver.cc:464] REPLICATION: finished
0504 13:56:23.625015 (+ 330us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0504 13:56:23.626576 (+ 1561us) log.cc:844] Serialized 5 byte log entry
0504 13:56:23.682445 (+ 55869us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:23.720962 (+ 38517us) log.cc:844] Serialized 14 byte log entry
0504 13:56:23.721666 (+ 704us) alter_schema_op.cc:73] Released schema lock
0504 13:56:23.721695 (+ 29us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":176,"prepare.queue_time_us":130,"prepare.run_cpu_time_us":41848,"prepare.run_wall_time_us":257146,"raft.queue_time_us":143,"raft.run_cpu_time_us":802,"raft.run_wall_time_us":801,"replication_time_us":1316634,"thread_start_us":258897,"threads_started":2,"wal-append.queue_time_us":263357,"wal-append.run_cpu_time_us":1224,"wal-append.run_wall_time_us":1005788}]]}
I20260504 13:56:23.727954 10497 raft_consensus.cc:493] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:23.728395 10497 raft_consensus.cc:515] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.730352 10494 raft_consensus.cc:493] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5beceb6317444247b31b44a6581707c6)
I20260504 13:56:23.730470 10497 leader_election.cc:290] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:23.730787 10494 raft_consensus.cc:515] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.732717 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50e129980cd84f6ca22cbf2addb06230" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:23.733079 10494 leader_election.cc:290] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:23.733271 9540 raft_consensus.cc:2468] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:23.734395 10013 tablet_bootstrap.cc:492] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:23.734776 10013 ts_tablet_manager.cc:1403] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.155s user 0.013s sys 0.007s
I20260504 13:56:23.735200 9767 leader_election.cc:304] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:23.736233 10494 raft_consensus.cc:2804] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260504 13:56:23.736269 9768 leader_election.cc:343] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 50e129980cd84f6ca22cbf2addb06230
I20260504 13:56:23.736581 10494 raft_consensus.cc:493] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:23.736884 10494 raft_consensus.cc:3060] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:23.737085 10013 raft_consensus.cc:359] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.737684 10013 raft_consensus.cc:385] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:23.737922 10013 raft_consensus.cc:740] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:23.738466 10013 consensus_queue.cc:260] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.743614 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd8cd1caaeb249e3bafee7f1f0d4df2f" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:23.745409 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_999e37d0dab54ca68c0fa66f74582109"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:23.746491 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd8cd1caaeb249e3bafee7f1f0d4df2f" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:23.747699 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_999e37d0dab54ca68c0fa66f74582109 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:23.747742 9767 leader_election.cc:304] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
W20260504 13:56:23.752278 9620 leader_election.cc:343] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:23.761752 10013 ts_tablet_manager.cc:1434] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.027s user 0.002s sys 0.003s
I20260504 13:56:23.782330 9337 master_service.cc:954] ReplaceTablet: received request to replace tablet 9ed4f5ea230a413a8dcdbb661ce173b6 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:23.787274 9344 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 0be795f04dd54cf19869c937b91d0ddf (table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]). Expected version 1 got 0
I20260504 13:56:23.788774 9344 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet b13303d2a14c4187b82da9f25ce9753e (table default.Table_b32f665dddb043769e07076397f84a73 [id=f227576f105c4b1fbd459b3638ce684b]). Expected version 1 got 0
I20260504 13:56:23.789750 9344 catalog_manager.cc:5671] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 1 to 2, leader changed from 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193) to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 2 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:23.796981 10494 raft_consensus.cc:515] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.795557 9944 ts_tablet_manager.cc:1929] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:23.797727 9944 log.cc:1199] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/c34877a81d504f46a2f680c5d5ddb4f4
I20260504 13:56:23.797909 10013 tablet_bootstrap.cc:492] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:23.798343 9944 ts_tablet_manager.cc:1950] T c34877a81d504f46a2f680c5d5ddb4f4 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:23.798755 10494 leader_election.cc:290] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:23.800786 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50e129980cd84f6ca22cbf2addb06230" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:23.801292 9539 raft_consensus.cc:3060] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:23.804823 9768 leader_election.cc:343] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 50e129980cd84f6ca22cbf2addb06230
I20260504 13:56:23.806715 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8db7c4732fa048f9950248f45d144c57" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:23.808024 9620 consensus_peers.cc:597] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:23.811782 9620 leader_election.cc:343] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:23.813401 9539 raft_consensus.cc:2468] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:23.794308 10497 raft_consensus.cc:2749] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:23.815096 9767 leader_election.cc:304] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:23.816735 10497 raft_consensus.cc:2804] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:23.817624 10497 raft_consensus.cc:697] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:23.818676 10429 raft_consensus.cc:493] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:23.818387 10497 consensus_queue.cc:237] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.819074 10429 raft_consensus.cc:515] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.822417 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1cf52a9e58744a99dae1958d2c1f867" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:23.823037 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1cf52a9e58744a99dae1958d2c1f867" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
W20260504 13:56:23.823592 9474 leader_election.cc:343] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:23.823633 9681 raft_consensus.cc:2468] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:23.824662 9473 leader_election.cc:304] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:23.825450 10431 raft_consensus.cc:2804] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:23.825784 10431 raft_consensus.cc:493] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:23.826098 10431 raft_consensus.cc:3060] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:23.836719 9931 tablet_bootstrap.cc:492] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:23.836841 10429 leader_election.cc:290] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:23.837162 9931 ts_tablet_manager.cc:1403] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.317s user 0.011s sys 0.004s
I20260504 13:56:23.839404 9657 tablet_service.cc:1558] Processing DeleteTablet for tablet 8e4e583afc804212bece2f727411cc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:23 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:23.839334 9931 raft_consensus.cc:359] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.839937 9931 raft_consensus.cc:385] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:23.840183 9931 raft_consensus.cc:740] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:23.843230 9807 tablet_service.cc:1511] Processing CreateTablet for tablet 4cf1504f0202481ba8b08f9bec5bc3d9 (DEFAULT_TABLE table=default.Table_737cc12f07144a6c92dc2974693a649d [id=e9103e8b725b4e81b6cf094395beef3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.844280 9807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf1504f0202481ba8b08f9bec5bc3d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.844980 9520 tablet_service.cc:1511] Processing CreateTablet for tablet 4cf1504f0202481ba8b08f9bec5bc3d9 (DEFAULT_TABLE table=default.Table_737cc12f07144a6c92dc2974693a649d [id=e9103e8b725b4e81b6cf094395beef3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.846107 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf1504f0202481ba8b08f9bec5bc3d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.855118 9657 tablet_service.cc:1511] Processing CreateTablet for tablet 4fae9274bb8240798f85b79504a227b3 (DEFAULT_TABLE table=default.Table_e116f9abbb23419d9a3365f7d0e17db9 [id=bee8acaa102249578514a3d4e5f876d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.856767 9657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fae9274bb8240798f85b79504a227b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.858884 10522 consensus_queue.cc:1048] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260504 13:56:23.859517 9618 leader_election.cc:343] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:23.859917 9618 leader_election.cc:304] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:23.860553 10522 raft_consensus.cc:2749] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:23.860999 9931 consensus_queue.cc:260] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.861595 10522 consensus_queue.cc:1048] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:23.862131 10429 consensus_queue.cc:1048] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:23.863600 9618 leader_election.cc:343] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:23.871976 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 145) took 2099 ms. Trace:
I20260504 13:56:23.872387 9944 rpcz_store.cc:276] 0504 13:56:21.772944 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.773658 (+ 714us) service_pool.cc:224] Handling call
0504 13:56:22.760618 (+986960us) ts_tablet_manager.cc:1151] Deleting tablet c34877a81d504f46a2f680c5d5ddb4f4
0504 13:56:22.760654 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:23.321097 (+560443us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:23.795293 (+474196us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:23.871899 (+ 76606us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":543317,"tablet-delete.queue_time_us":986405}
I20260504 13:56:23.873934 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet c34877a81d504f46a2f680c5d5ddb4f4 (table default.Table_c6a898ba23b74b54bb4684e970b44742 [id=92ea89acf1fc427690ee0a61cbbb1647]) successfully deleted
I20260504 13:56:23.875017 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 146) took 2096 ms. Trace:
I20260504 13:56:23.903872 9806 tablet_service.cc:1511] Processing CreateTablet for tablet 8e4e583afc804212bece2f727411cc14 (DEFAULT_TABLE table=default.Table_a8bc83249ad949c49e759a07818e2c02 [id=4182fad7cb2344d88498c26192164a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.905006 9806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e4e583afc804212bece2f727411cc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.911270 9337 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 9ed4f5ea230a413a8dcdbb661ce173b6
I20260504 13:56:23.911275 9944 rpcz_store.cc:276] 0504 13:56:21.779059 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.779288 (+ 229us) service_pool.cc:224] Handling call
0504 13:56:23.873671 (+2094383us) ts_tablet_manager.cc:1151] Deleting tablet eb7f58645c76441589368072274ebc8f
0504 13:56:23.874787 (+ 1116us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:23.874959 (+ 172us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2093840}
W20260504 13:56:23.912225 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet eb7f58645c76441589368072274ebc8f because the tablet was not found. No further retry: Not found: Tablet not found: eb7f58645c76441589368072274ebc8f
I20260504 13:56:23.880770 9658 tablet_service.cc:1511] Processing CreateTablet for tablet 4cf1504f0202481ba8b08f9bec5bc3d9 (DEFAULT_TABLE table=default.Table_737cc12f07144a6c92dc2974693a649d [id=e9103e8b725b4e81b6cf094395beef3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.913432 9337 catalog_manager.cc:6741] ReplaceTablet: tablet 9ed4f5ea230a413a8dcdbb661ce173b6 deleted and replaced by tablet 88619eaa917d4c33a910fc1347ab126e
I20260504 13:56:23.913511 9665 tablet_service.cc:1558] Processing DeleteTablet for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 88619eaa917d4c33a910fc1347ab126e) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:23.914147 9658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf1504f0202481ba8b08f9bec5bc3d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.916414 9342 catalog_manager.cc:5671] T ab4c185332a24d96bc3d0fddc0d20693 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:23.863969 9618 leader_election.cc:304] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:23.905912 9656 tablet_service.cc:1511] Processing CreateTablet for tablet 8bb77106b48a45a2aaa938dc474fa8bd (DEFAULT_TABLE table=default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.914427 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_e55147137c794be4bff21afce5d36bbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:23.919840 9656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bb77106b48a45a2aaa938dc474fa8bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.920534 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 88619eaa917d4c33a910fc1347ab126e) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:23.921202 9507 tablet_service.cc:1558] Processing DeleteTablet for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 88619eaa917d4c33a910fc1347ab126e) from {username='slave'} at 127.0.0.1:52642
W20260504 13:56:23.922088 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e55147137c794be4bff21afce5d36bbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:23.922531 9508 tablet_service.cc:1511] Processing CreateTablet for tablet 8e4e583afc804212bece2f727411cc14 (DEFAULT_TABLE table=default.Table_a8bc83249ad949c49e759a07818e2c02 [id=4182fad7cb2344d88498c26192164a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.906409 9659 tablet_service.cc:1511] Processing CreateTablet for tablet 8e4e583afc804212bece2f727411cc14 (DEFAULT_TABLE table=default.Table_a8bc83249ad949c49e759a07818e2c02 [id=4182fad7cb2344d88498c26192164a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.924098 9659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e4e583afc804212bece2f727411cc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.924096 10496 tablet_replica.cc:333] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:23.923966 9518 tablet_service.cc:1511] Processing CreateTablet for tablet 8bb77106b48a45a2aaa938dc474fa8bd (DEFAULT_TABLE table=default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:23.924952 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bb77106b48a45a2aaa938dc474fa8bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.924935 10496 raft_consensus.cc:2243] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:23.925467 10496 raft_consensus.cc:2272] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:23.927925 9508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e4e583afc804212bece2f727411cc14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:23.928159 10496 ts_tablet_manager.cc:1916] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:23.931535 10393 raft_consensus.cc:2749] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:23.932379 9618 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 68813us
I20260504 13:56:23.891669 10431 raft_consensus.cc:515] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.936854 9342 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 0be795f04dd54cf19869c937b91d0ddf (table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]). Expected version 1 got 0
I20260504 13:56:23.937924 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1cf52a9e58744a99dae1958d2c1f867" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:23.938985 9342 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 604f1ba5f71041a7877daff8972a994a (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]). Expected version 2 got 0
W20260504 13:56:23.939563 9474 leader_election.cc:343] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:23.940028 9342 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet b13303d2a14c4187b82da9f25ce9753e (table default.Table_b32f665dddb043769e07076397f84a73 [id=f227576f105c4b1fbd459b3638ce684b]). Expected version 1 got 0
I20260504 13:56:23.942013 10431 leader_election.cc:290] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:23.942358 9678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1cf52a9e58744a99dae1958d2c1f867" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:23.948206 9678 raft_consensus.cc:3060] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:23.959384 9931 ts_tablet_manager.cc:1434] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.122s user 0.007s sys 0.001s
I20260504 13:56:23.960246 9931 tablet_bootstrap.cc:492] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:23.965624 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8db7c4732fa048f9950248f45d144c57" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:23.976617 10382 raft_consensus.cc:493] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:23.977156 10382 raft_consensus.cc:515] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.980649 10382 leader_election.cc:290] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:23.982280 9768 leader_election.cc:343] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: afad9329cf7745ba8168b2a93f998295
I20260504 13:56:23.984267 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afad9329cf7745ba8168b2a93f998295" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:23.984963 9678 raft_consensus.cc:2468] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:23.985066 9539 raft_consensus.cc:2468] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:23.985826 9473 leader_election.cc:304] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:23.986224 9767 leader_election.cc:304] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:23.987087 10431 raft_consensus.cc:2804] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:23.987556 10431 raft_consensus.cc:697] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:23.987608 10382 raft_consensus.cc:2804] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:23.988018 10382 raft_consensus.cc:493] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:23.988394 10382 raft_consensus.cc:3060] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:23.988500 10431 consensus_queue.cc:237] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:23.997058 9944 tablet_replica.cc:333] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:24.001758 9931 tablet_bootstrap.cc:654] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:24.015352 9944 raft_consensus.cc:2243] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:24.029664 9337 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_2abb0f27da024032b5646af14e0946e8" } modify_external_catalogs: true
I20260504 13:56:24.030329 9337 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_2abb0f27da024032b5646af14e0946e8" } modify_external_catalogs: true
I20260504 13:56:24.034273 9944 raft_consensus.cc:2272] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:24.035420 9618 leader_election.cc:343] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:24.035781 9618 leader_election.cc:304] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:24.031620 9932 tablet_bootstrap.cc:492] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:24.036808 9932 ts_tablet_manager.cc:1403] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.499s user 0.011s sys 0.000s
I20260504 13:56:24.041491 9944 ts_tablet_manager.cc:1916] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:24.042924 10522 raft_consensus.cc:2749] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:24.045881 9932 raft_consensus.cc:359] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.046504 9932 raft_consensus.cc:385] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:24.046761 9932 raft_consensus.cc:740] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:24.053052 9806 tablet_service.cc:1511] Processing CreateTablet for tablet 8bb77106b48a45a2aaa938dc474fa8bd (DEFAULT_TABLE table=default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.054090 9806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bb77106b48a45a2aaa938dc474fa8bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.058542 10382 raft_consensus.cc:515] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.060119 10496 ts_tablet_manager.cc:1929] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:24.060508 10496 log.cc:1199] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/9ed4f5ea230a413a8dcdbb661ce173b6
I20260504 13:56:24.061364 10496 ts_tablet_manager.cc:1950] T 9ed4f5ea230a413a8dcdbb661ce173b6 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:24.062784 9807 tablet_service.cc:1511] Processing CreateTablet for tablet e0b9e47b0a4443c5830422b9d0553446 (DEFAULT_TABLE table=default.Table_4207c9e8019c4088b934376c5568d101 [id=3ca89308de104a5fa27cac4c62047304]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.063872 9807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0b9e47b0a4443c5830422b9d0553446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.064244 9932 consensus_queue.cc:260] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.067566 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]) successfully deleted
I20260504 13:56:24.071070 9932 ts_tablet_manager.cc:1434] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20260504 13:56:24.072011 10393 raft_consensus.cc:493] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:24.072429 10393 raft_consensus.cc:515] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.073947 10522 raft_consensus.cc:493] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:24.074415 9340 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 0be795f04dd54cf19869c937b91d0ddf (table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]). Expected version 1 got 0
I20260504 13:56:24.074368 10522 raft_consensus.cc:515] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.076040 10522 leader_election.cc:290] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:24.077399 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fbe09b4078c4eebb52d7d00bc87371c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:24.079823 10393 leader_election.cc:290] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:24.079923 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8989cdde4ca4e349079596e33723b2c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:24.081091 9620 leader_election.cc:343] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:24.081638 9620 leader_election.cc:343] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:24.075644 9340 catalog_manager.cc:5671] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:24.086041 9340 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet b13303d2a14c4187b82da9f25ce9753e (table default.Table_b32f665dddb043769e07076397f84a73 [id=f227576f105c4b1fbd459b3638ce684b]). Expected version 1 got 0
I20260504 13:56:24.086494 9932 tablet_bootstrap.cc:492] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:24.090528 10382 leader_election.cc:290] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:24.092173 9768 leader_election.cc:343] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: afad9329cf7745ba8168b2a93f998295
I20260504 13:56:24.081094 10013 tablet_bootstrap.cc:654] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:24.018803 10391 ts_tablet_manager.cc:1929] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:24.099014 9931 tablet_bootstrap.cc:492] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:24.099460 9931 ts_tablet_manager.cc:1403] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.139s user 0.006s sys 0.003s
I20260504 13:56:24.105062 9931 raft_consensus.cc:359] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.105659 9931 raft_consensus.cc:385] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:24.105912 9931 raft_consensus.cc:740] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:24.109920 9520 tablet_service.cc:1511] Processing CreateTablet for tablet e0b9e47b0a4443c5830422b9d0553446 (DEFAULT_TABLE table=default.Table_4207c9e8019c4088b934376c5568d101 [id=3ca89308de104a5fa27cac4c62047304]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.116309 9340 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:24.117669 9340 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 8db7c4732fa048f9950248f45d144c57 (table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62]). Expected version 1 got 0
I20260504 13:56:24.118208 9932 tablet_bootstrap.cc:654] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:24.120702 9340 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 1fbe09b4078c4eebb52d7d00bc87371c (table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee]). Expected version 1 got 0
I20260504 13:56:24.121790 9944 ts_tablet_manager.cc:1929] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:24.122172 9944 log.cc:1199] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/bd8cd1caaeb249e3bafee7f1f0d4df2f
I20260504 13:56:24.122879 9944 ts_tablet_manager.cc:1950] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:24.125922 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 147) took 2345 ms. Trace:
I20260504 13:56:24.126240 9944 rpcz_store.cc:276] 0504 13:56:21.780126 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.780238 (+ 112us) service_pool.cc:224] Handling call
0504 13:56:23.996899 (+2216661us) ts_tablet_manager.cc:1151] Deleting tablet bd8cd1caaeb249e3bafee7f1f0d4df2f
0504 13:56:23.996943 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:24.097782 (+100839us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:24.121666 (+ 23884us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:24.125871 (+ 4205us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2216183}
I20260504 13:56:24.127383 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet bd8cd1caaeb249e3bafee7f1f0d4df2f (table default.Table_47c9ed0b601146b687a576ec89c8b626 [id=b75b9e5edced4443944af4087b77e54d]) successfully deleted
I20260504 13:56:24.129580 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 148) took 2023 ms. Trace:
I20260504 13:56:24.129868 9944 rpcz_store.cc:276] 0504 13:56:22.106109 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:22.106220 (+ 111us) service_pool.cc:224] Handling call
0504 13:56:24.129360 (+2023140us) ts_tablet_manager.cc:1151] Deleting tablet 65052a5c28c147a385144317e21eea41
0504 13:56:24.129398 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:24.129533 (+ 135us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2022175}
W20260504 13:56:24.130918 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 65052a5c28c147a385144317e21eea41 because the tablet was not found. No further retry: Not found: Tablet not found: 65052a5c28c147a385144317e21eea41
I20260504 13:56:24.132331 10393 consensus_queue.cc:1048] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:24.122835 9340 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 604f1ba5f71041a7877daff8972a994a (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]). Expected version 2 got 0
I20260504 13:56:24.146605 9340 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]). Expected version 2 got 0
I20260504 13:56:24.147357 9335 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 1501455a0af348e38746d100987afdd0
W20260504 13:56:24.147795 9339 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b13303d2a14c4187b82da9f25ce9753e consensus state has no leader
I20260504 13:56:24.148190 9341 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35768 (request call id 226) took 1406 ms. Trace:
I20260504 13:56:24.148236 9339 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35768 (request call id 231) took 1212 ms. Trace:
I20260504 13:56:24.148612 9335 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35768 (request call id 223) took 1608 ms. Trace:
I20260504 13:56:24.148627 9339 rpcz_store.cc:276] 0504 13:56:22.935792 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:22.935904 (+ 112us) service_pool.cc:224] Handling call
0504 13:56:22.936755 (+ 851us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:22.937524 (+ 769us) catalog_manager.cc:3725] Apply alter schema
0504 13:56:22.938040 (+ 516us) catalog_manager.cc:3757] Acquired catalog manager lock
0504 13:56:22.938209 (+ 169us) catalog_manager.cc:3915] Updating metadata on disk
0504 13:56:23.013441 (+ 75232us) catalog_manager.cc:3946] Committing alterations to in-memory state
0504 13:56:24.148188 (+1134747us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:22.939363 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:22.939723 (+ 360us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:22.939745 (+ 22us) write_op.cc:435] Acquired schema lock
0504 13:56:22.939758 (+ 13us) tablet.cc:662] Decoding operations
0504 13:56:22.939976 (+ 218us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:22.940009 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:22.940029 (+ 20us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:22.940157 (+ 128us) tablet.cc:701] Row locks acquired
0504 13:56:22.940169 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:22.940259 (+ 90us) write_op.cc:270] Start()
0504 13:56:22.940325 (+ 66us) write_op.cc:276] Timestamp: P: 1777902982940217 usec, L: 0
0504 13:56:22.940340 (+ 15us) op_driver.cc:348] REPLICATION: starting
0504 13:56:22.962571 (+ 22231us) log.cc:844] Serialized 405 byte log entry
0504 13:56:22.964532 (+ 1961us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b801951 00007fc27b894208 00007fc27bd069f1 00007fc27bd061c8 00007fc27bd0ae61 00007fc27bd0ae19 00007fc27bd0ada9 00007fc27bd0ad71 00007fc27bd09f9d 00000000005b3bc4 00000000005b39f9
0504 13:56:22.971296 (+ 6764us) op_driver.cc:464] REPLICATION: finished
0504 13:56:22.978884 (+ 7588us) write_op.cc:301] APPLY: starting
0504 13:56:22.978938 (+ 54us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:22.979024 (+ 86us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:22.979047 (+ 23us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:22.979240 (+ 193us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:22.979382 (+ 142us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:22.979422 (+ 40us) write_op.cc:312] APPLY: finished
0504 13:56:22.979587 (+ 165us) log.cc:844] Serialized 23 byte log entry
0504 13:56:22.979733 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:22.979794 (+ 61us) write_op.cc:454] Released schema lock
0504 13:56:22.979820 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:23.014863 (+ 35043us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:23.014897 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":7435,"apply.run_cpu_time_us":1153,"apply.run_wall_time_us":34923,"num_ops":1,"prepare.queue_time_us":128,"prepare.run_cpu_time_us":2541,"prepare.run_wall_time_us":26329,"raft.queue_time_us":94,"raft.run_cpu_time_us":767,"raft.run_wall_time_us":768,"replication_time_us":30873,"spinlock_wait_cycles":49995520}]]}
I20260504 13:56:24.106465 9931 consensus_queue.cc:260] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.157470 9944 tablet_replica.cc:333] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:24.159838 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_bff621bd8f674f7fa3cdb3de52bb8386"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:24.161070 9932 tablet_bootstrap.cc:492] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:24.148916 9335 rpcz_store.cc:276] 0504 13:56:22.539941 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:22.540066 (+ 125us) service_pool.cc:224] Handling call
0504 13:56:22.541413 (+ 1347us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:22.542109 (+ 696us) catalog_manager.cc:2815] Modifying in-memory table state
0504 13:56:22.542237 (+ 128us) catalog_manager.cc:2824] Locking tablets
0504 13:56:23.295314 (+753077us) catalog_manager.cc:2838] Removing table and tablets from system table
0504 13:56:23.656517 (+361203us) catalog_manager.cc:2856] Removing table from by-name map
0504 13:56:23.656710 (+ 193us) catalog_manager.cc:2872] Committing in-memory state
0504 13:56:24.147282 (+490572us) catalog_manager.cc:2876] Aborting table tasks
0504 13:56:24.148543 (+ 1261us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:23.446445 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:23.446850 (+ 405us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:23.446874 (+ 24us) write_op.cc:435] Acquired schema lock
0504 13:56:23.446888 (+ 14us) tablet.cc:662] Decoding operations
0504 13:56:23.495306 (+ 48418us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:23.495354 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:23.495386 (+ 32us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:23.495666 (+ 280us) tablet.cc:701] Row locks acquired
0504 13:56:23.495682 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:23.495796 (+ 114us) write_op.cc:270] Start()
0504 13:56:23.497528 (+ 1732us) write_op.cc:276] Timestamp: P: 1777902983495749 usec, L: 0
0504 13:56:23.497551 (+ 23us) op_driver.cc:348] REPLICATION: starting
0504 13:56:23.498066 (+ 515us) log.cc:844] Serialized 731 byte log entry
0504 13:56:23.654416 (+156350us) op_driver.cc:464] REPLICATION: finished
0504 13:56:23.654805 (+ 389us) write_op.cc:301] APPLY: starting
0504 13:56:23.654859 (+ 54us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:23.654965 (+ 106us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:23.655057 (+ 92us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:23.655661 (+ 604us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:23.655829 (+ 168us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:23.655882 (+ 53us) write_op.cc:312] APPLY: finished
0504 13:56:23.656073 (+ 191us) log.cc:844] Serialized 29 byte log entry
0504 13:56:23.656221 (+ 148us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:23.656286 (+ 65us) write_op.cc:454] Released schema lock
0504 13:56:23.656321 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:23.656546 (+ 225us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:23.656569 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":63,"child_traces":[["op",{"apply.queue_time_us":264,"apply.run_cpu_time_us":1384,"apply.run_wall_time_us":1704,"num_ops":2,"prepare.queue_time_us":149449,"prepare.run_cpu_time_us":2332,"prepare.run_wall_time_us":54077,"raft.queue_time_us":125,"raft.run_cpu_time_us":584,"raft.run_wall_time_us":865,"replication_time_us":156758}]]}
I20260504 13:56:24.161473 9932 ts_tablet_manager.cc:1403] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.075s user 0.015s sys 0.000s
I20260504 13:56:24.163100 9335 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_3387191fcd6046abbf257447bfbb27f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:24.164598 9944 raft_consensus.cc:2243] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Raft consensus shutting down.
W20260504 13:56:24.165777 9335 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3387191fcd6046abbf257447bfbb27f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:24.149636 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet 1501455a0af348e38746d100987afdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:24.166004 9944 raft_consensus.cc:2272] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:24.164690 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bff621bd8f674f7fa3cdb3de52bb8386 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:24.169629 10496 tablet_replica.cc:333] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:24.148494 9341 rpcz_store.cc:276] 0504 13:56:22.741773 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:22.741922 (+ 149us) service_pool.cc:224] Handling call
0504 13:56:22.743074 (+ 1152us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:22.744056 (+ 982us) catalog_manager.cc:3725] Apply alter schema
0504 13:56:22.744716 (+ 660us) catalog_manager.cc:3757] Acquired catalog manager lock
0504 13:56:22.745044 (+ 328us) catalog_manager.cc:3915] Updating metadata on disk
0504 13:56:22.777934 (+ 32890us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:22.777968 (+ 34us) write_op.cc:454] Released schema lock
0504 13:56:22.778199 (+ 231us) catalog_manager.cc:3946] Committing alterations to in-memory state
0504 13:56:24.148138 (+1369939us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:22.746464 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:22.746943 (+ 479us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:22.746976 (+ 33us) write_op.cc:435] Acquired schema lock
0504 13:56:22.746995 (+ 19us) tablet.cc:662] Decoding operations
0504 13:56:22.747298 (+ 303us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:22.747342 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:22.747374 (+ 32us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:22.747555 (+ 181us) tablet.cc:701] Row locks acquired
0504 13:56:22.747572 (+ 17us) write_op.cc:260] PREPARE: finished
0504 13:56:22.747687 (+ 115us) write_op.cc:270] Start()
0504 13:56:22.747783 (+ 96us) write_op.cc:276] Timestamp: P: 1777902982747619 usec, L: 0
0504 13:56:22.747804 (+ 21us) op_driver.cc:348] REPLICATION: starting
0504 13:56:22.748467 (+ 663us) log.cc:844] Serialized 405 byte log entry
0504 13:56:22.756939 (+ 8472us) op_driver.cc:464] REPLICATION: finished
0504 13:56:22.758062 (+ 1123us) write_op.cc:301] APPLY: starting
0504 13:56:22.758133 (+ 71us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:22.758237 (+ 104us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:22.758353 (+ 116us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:22.758525 (+ 172us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:22.758685 (+ 160us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:22.758744 (+ 59us) write_op.cc:312] APPLY: finished
0504 13:56:22.758943 (+ 199us) log.cc:844] Serialized 23 byte log entry
0504 13:56:22.759099 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:22.759194 (+ 95us) write_op.cc:454] Released schema lock
0504 13:56:22.759234 (+ 40us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":953,"apply.run_cpu_time_us":1373,"apply.run_wall_time_us":1372,"num_ops":1,"prepare.queue_time_us":104,"prepare.run_cpu_time_us":2250,"prepare.run_wall_time_us":26473,"raft.queue_time_us":87,"raft.run_cpu_time_us":640,"raft.run_wall_time_us":641,"replication_time_us":9025}]]}
I20260504 13:56:24.170619 10496 raft_consensus.cc:2243] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:24.171159 10496 raft_consensus.cc:2272] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:24.172025 9341 master_service.cc:954] ReplaceTablet: received request to replace tablet 604f1ba5f71041a7877daff8972a994a from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:24.174043 10496 ts_tablet_manager.cc:1916] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:24.182472 9344 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: default.Table_91cc6c38c9ff43dca25814664068b1c6 [id=da5c427e75e74ea1ae3a5961ed27e0de] alter complete (version 1)
I20260504 13:56:24.185328 9931 ts_tablet_manager.cc:1434] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.086s user 0.005s sys 0.000s
I20260504 13:56:24.188604 9944 ts_tablet_manager.cc:1916] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:24.186193 9931 tablet_bootstrap.cc:492] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:24.192171 9932 raft_consensus.cc:359] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.194397 9932 raft_consensus.cc:385] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:24.194653 9932 raft_consensus.cc:740] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:24.197444 9932 consensus_queue.cc:260] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.202513 9339 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_655e2e8cf3544fbeb0ce33829bb2c8a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:24.209188 10393 raft_consensus.cc:493] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:24.209599 10393 raft_consensus.cc:515] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.211705 10393 leader_election.cc:290] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
W20260504 13:56:24.215732 9339 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_655e2e8cf3544fbeb0ce33829bb2c8a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:24.220379 9337 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet baf07a970a7c4a35b3c849b06d6c88d0
I20260504 13:56:24.221299 9932 ts_tablet_manager.cc:1434] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.060s user 0.006s sys 0.000s
I20260504 13:56:24.230623 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet baf07a970a7c4a35b3c849b06d6c88d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:24 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:24.231684 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_583272d3574a4ca78d056fe7d6b922bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:24.233103 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_583272d3574a4ca78d056fe7d6b922bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:24.239892 9341 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 604f1ba5f71041a7877daff8972a994a
I20260504 13:56:24.243407 9341 catalog_manager.cc:6741] ReplaceTablet: tablet 604f1ba5f71041a7877daff8972a994a deleted and replaced by tablet 2ddf890c17964e7f90409ebe3c4034ef
I20260504 13:56:24.120790 10391 log.cc:1199] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/70caac2e256d419dbc2e70cb5dfffe97
I20260504 13:56:24.128554 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afad9329cf7745ba8168b2a93f998295" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:24.131809 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fbe09b4078c4eebb52d7d00bc87371c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:24.131839 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8989cdde4ca4e349079596e33723b2c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:24.248142 9618 leader_election.cc:343] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:24.248299 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet 604f1ba5f71041a7877daff8972a994a with delete_type TABLET_DATA_DELETED (replaced by tablet 2ddf890c17964e7f90409ebe3c4034ef) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:24.248570 9618 leader_election.cc:304] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:24.248888 9678 raft_consensus.cc:1275] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 5beceb6317444247b31b44a6581707c6 [term 2 FOLLOWER]: Refusing update from remote peer d4051d2bf0bf410eb8acd107560df888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260504 13:56:24.249336 10513 raft_consensus.cc:2749] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:24.181073 9511 tablet_service.cc:1558] Processing DeleteTablet for tablet 1501455a0af348e38746d100987afdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:24.219291 10429 raft_consensus.cc:493] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:24.221958 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f3905bfb30e4b03a7fcf0a2f57e2117" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:24.250190 10382 consensus_queue.cc:1048] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:24.251611 9932 tablet_bootstrap.cc:492] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:24.260743 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f3905bfb30e4b03a7fcf0a2f57e2117" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:24.268601 9931 tablet_bootstrap.cc:654] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:24.257535 9665 tablet_service.cc:1558] Processing DeleteTablet for tablet 604f1ba5f71041a7877daff8972a994a with delete_type TABLET_DATA_DELETED (replaced by tablet 2ddf890c17964e7f90409ebe3c4034ef) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:24.262194 9656 tablet_service.cc:1558] Processing DeleteTablet for tablet baf07a970a7c4a35b3c849b06d6c88d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:24 UTC) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:24.263110 9620 leader_election.cc:343] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:24.223455 9507 tablet_service.cc:1558] Processing DeleteTablet for tablet baf07a970a7c4a35b3c849b06d6c88d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:24 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:24.274502 9656 tablet_service.cc:1511] Processing CreateTablet for tablet e0b9e47b0a4443c5830422b9d0553446 (DEFAULT_TABLE table=default.Table_4207c9e8019c4088b934376c5568d101 [id=3ca89308de104a5fa27cac4c62047304]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.275807 9656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0b9e47b0a4443c5830422b9d0553446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.276099 10499 tablet.cc:1724] T b13303d2a14c4187b82da9f25ce9753e P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260504 13:56:24.284060 9336 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:55533 (request call id 40) took 1860 ms (client timeout 1000 ms). Trace:
I20260504 13:56:24.288825 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet c34877a81d504f46a2f680c5d5ddb4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 359b0fa01a1648a7bef6f97960a7b9a8) from {username='slave'} at 127.0.0.1:56380
W20260504 13:56:24.229719 9474 consensus_peers.cc:597] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ed4f5ea230a413a8dcdbb661ce173b6. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:24.291345 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 61723.1us
I20260504 13:56:24.241272 9506 tablet_service.cc:1558] Processing DeleteTablet for tablet 604f1ba5f71041a7877daff8972a994a with delete_type TABLET_DATA_DELETED (replaced by tablet 2ddf890c17964e7f90409ebe3c4034ef) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:24.250931 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0b9e47b0a4443c5830422b9d0553446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.298239 9663 tablet_service.cc:1558] Processing DeleteTablet for tablet c34630d55f4a4e19b33ddf4c13995a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:19 UTC) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:24.284375 9336 rpcz_store.cc:269] 0504 13:56:22.423326 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:22.432545 (+ 9219us) service_pool.cc:224] Handling call
0504 13:56:24.283973 (+1851428us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:23.207835 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:23.208203 (+ 368us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:23.208226 (+ 23us) write_op.cc:435] Acquired schema lock
0504 13:56:23.208239 (+ 13us) tablet.cc:662] Decoding operations
0504 13:56:23.208481 (+ 242us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:23.208515 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:23.208536 (+ 21us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:23.208665 (+ 129us) tablet.cc:701] Row locks acquired
0504 13:56:23.208680 (+ 15us) write_op.cc:260] PREPARE: finished
0504 13:56:23.208795 (+ 115us) write_op.cc:270] Start()
0504 13:56:23.208884 (+ 89us) write_op.cc:276] Timestamp: P: 1777902983208728 usec, L: 0
0504 13:56:23.208899 (+ 15us) op_driver.cc:348] REPLICATION: starting
0504 13:56:23.209350 (+ 451us) log.cc:844] Serialized 533 byte log entry
0504 13:56:23.238102 (+ 28752us) op_driver.cc:464] REPLICATION: finished
0504 13:56:23.239001 (+ 899us) write_op.cc:301] APPLY: starting
0504 13:56:23.239048 (+ 47us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:23.243530 (+ 4482us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:23.243626 (+ 96us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:23.243767 (+ 141us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:23.243888 (+ 121us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:23.243928 (+ 40us) write_op.cc:312] APPLY: finished
0504 13:56:23.244063 (+ 135us) log.cc:844] Serialized 23 byte log entry
0504 13:56:23.244174 (+ 111us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:23.244225 (+ 51us) write_op.cc:454] Released schema lock
0504 13:56:23.244250 (+ 25us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:23.251386 (+ 7136us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:23.251422 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":777,"apply.run_cpu_time_us":1077,"apply.run_wall_time_us":12334,"num_ops":1,"prepare.queue_time_us":209079,"prepare.run_cpu_time_us":1761,"prepare.run_wall_time_us":1761,"raft.queue_time_us":909,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":11,"replication_time_us":29130}]]}
I20260504 13:56:24.299928 9665 tablet_service.cc:1558] Processing DeleteTablet for tablet c34877a81d504f46a2f680c5d5ddb4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 359b0fa01a1648a7bef6f97960a7b9a8) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:24.307475 9665 tablet_service.cc:1558] Processing DeleteTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:16 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:24.270366 9536 raft_consensus.cc:3060] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:24.315140 10391 ts_tablet_manager.cc:1950] T 70caac2e256d419dbc2e70cb5dfffe97 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:24.291795 10429 raft_consensus.cc:515] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:24.335330 9809 tablet_service.cc:1511] Processing CreateTablet for tablet f3e86027e3e74bc196e9a9ffc9fdf8a5 (DEFAULT_TABLE table=default.Table_780e641042db4fd5aeb8af8b1aa08c72 [id=6cb26719d5fe45ccbdef18571c902228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.336495 9809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e86027e3e74bc196e9a9ffc9fdf8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.341743 10429 leader_election.cc:290] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:24.357378 9536 raft_consensus.cc:2468] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
W20260504 13:56:24.358072 9338 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 33) took 2342 ms (client timeout 1000 ms). Trace:
W20260504 13:56:24.358379 9338 rpcz_store.cc:269] 0504 13:56:22.015876 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:22.130948 (+115072us) service_pool.cc:224] Handling call
0504 13:56:22.443617 (+312669us) spinlock_profiling.cc:243] Waited 30.8 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:24.358011 (+1914394us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":23,"spinlock_wait_cycles":70761728}
I20260504 13:56:24.359573 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 133) took 3094 ms. Trace:
I20260504 13:56:24.359985 10391 rpcz_store.cc:276] 0504 13:56:21.265581 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.296884 (+ 31303us) service_pool.cc:224] Handling call
0504 13:56:23.603699 (+2306815us) ts_tablet_manager.cc:1151] Deleting tablet 70caac2e256d419dbc2e70cb5dfffe97
0504 13:56:23.603743 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:23.754786 (+151043us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:24.018653 (+263867us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:24.359521 (+340868us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64644,"tablet-delete.queue_time_us":2306217}
I20260504 13:56:24.361390 10391 tablet_replica.cc:333] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:24.362289 10391 raft_consensus.cc:2243] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:24.362807 10391 raft_consensus.cc:2272] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:24.365958 10391 ts_tablet_manager.cc:1916] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:24.366858 9814 tablet_service.cc:1511] Processing CreateTablet for tablet de6714ee86d64112baf13dd1b2b87582 (DEFAULT_TABLE table=default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.368032 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de6714ee86d64112baf13dd1b2b87582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.380817 9804 tablet_service.cc:1511] Processing CreateTablet for tablet ba4584755aae47e0b0f14c6c55474d18 (DEFAULT_TABLE table=default.Table_2d5a582adb634895a55193dd861f3b60 [id=30f349197d564e8da82855fec17b4cd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.381892 9804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba4584755aae47e0b0f14c6c55474d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.379861 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "604f1ba5f71041a7877daff8972a994a" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:24.370362 10013 tablet_bootstrap.cc:492] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
W20260504 13:56:24.375933 9474 consensus_peers.cc:597] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:24.394841 10013 ts_tablet_manager.cc:1403] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.597s user 0.010s sys 0.004s
I20260504 13:56:24.388055 9767 leader_election.cc:304] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:24.405897 9811 tablet_service.cc:1511] Processing CreateTablet for tablet 2f6b3e969760442283c3191e062c1839 (DEFAULT_TABLE table=default.Table_5bc32a610edd49ab9641c3726f53f5c6 [id=be5c761518674e7782bfff73366b1675]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.407088 9811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f6b3e969760442283c3191e062c1839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.415266 9805 tablet_service.cc:1511] Processing CreateTablet for tablet c47b6e3b4d0c45c28384f7aae6b749de (DEFAULT_TABLE table=default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.416543 9805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c47b6e3b4d0c45c28384f7aae6b749de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.428253 9807 tablet_service.cc:1511] Processing CreateTablet for tablet a26c6a2ffb8746a3941a56c7259412e3 (DEFAULT_TABLE table=default.Table_225ad80b8318432b88a729064c88e8ec [id=a3cca003025745cebf8e7d403d0778cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.429600 9807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a26c6a2ffb8746a3941a56c7259412e3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:24.433708 9474 leader_election.cc:343] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:24.422051 10382 raft_consensus.cc:2804] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:24.449472 9810 tablet_service.cc:1511] Processing CreateTablet for tablet 2a629c2a69084b6b98984c8d374eeb1e (DEFAULT_TABLE table=default.Table_a5fa0a0bead94aa3b02d6175fefde171 [id=e9fbd43d30ae4d1dbeea534025125b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.450512 9810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a629c2a69084b6b98984c8d374eeb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.469529 10382 raft_consensus.cc:697] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:24.470429 10382 consensus_queue.cc:237] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:24.476356 9618 leader_election.cc:343] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:24.477167 9618 leader_election.cc:304] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:24.478735 9618 leader_election.cc:343] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:24.479193 9618 leader_election.cc:304] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:24.480207 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 179326us
I20260504 13:56:24.486408 10013 raft_consensus.cc:359] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.487046 10013 raft_consensus.cc:385] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:24.487365 10013 raft_consensus.cc:740] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:24.488098 10496 ts_tablet_manager.cc:1929] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:24.487931 10013 consensus_queue.cc:260] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.491856 9518 tablet_service.cc:1511] Processing CreateTablet for tablet ba4584755aae47e0b0f14c6c55474d18 (DEFAULT_TABLE table=default.Table_2d5a582adb634895a55193dd861f3b60 [id=30f349197d564e8da82855fec17b4cd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.492833 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba4584755aae47e0b0f14c6c55474d18. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:24.499647 9331 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.254:36145, user_credentials={real_user=slave}} blocked reactor thread for 151576us
I20260504 13:56:24.488593 10496 log.cc:1199] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/1501455a0af348e38746d100987afdd0
I20260504 13:56:24.502203 10496 ts_tablet_manager.cc:1950] T 1501455a0af348e38746d100987afdd0 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:24.503320 10496 tablet_replica.cc:333] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:24.504128 10496 raft_consensus.cc:2243] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:24.504643 10496 raft_consensus.cc:2272] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:24.505782 10562 tablet.cc:1724] T b13303d2a14c4187b82da9f25ce9753e P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:24.512339 9508 tablet_service.cc:1511] Processing CreateTablet for tablet de6714ee86d64112baf13dd1b2b87582 (DEFAULT_TABLE table=default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.513374 9508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de6714ee86d64112baf13dd1b2b87582. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:24.518656 9657 consensus_meta.cc:287] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6: Time spent flushing consensus metadata: real 0.609s user 0.013s sys 0.000s
I20260504 13:56:24.520304 9510 tablet_service.cc:1511] Processing CreateTablet for tablet f3e86027e3e74bc196e9a9ffc9fdf8a5 (DEFAULT_TABLE table=default.Table_780e641042db4fd5aeb8af8b1aa08c72 [id=6cb26719d5fe45ccbdef18571c902228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.520955 9806 tablet_service.cc:1511] Processing CreateTablet for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (DEFAULT_TABLE table=default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.521389 9510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e86027e3e74bc196e9a9ffc9fdf8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.522171 9806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.525374 9932 tablet_bootstrap.cc:654] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:24.525108 9658 tablet_service.cc:1558] Processing DeleteTablet for tablet 1501455a0af348e38746d100987afdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:24.526314 9808 tablet_service.cc:1511] Processing CreateTablet for tablet 4a1a77ad56634e068401a2bfbc977761 (DEFAULT_TABLE table=default.Table_9bc232a8e4dd4df1a36b35d09dd50d0a [id=66faab34fb324213a874ea6fae3c96a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.527246 9808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a1a77ad56634e068401a2bfbc977761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.532246 9506 tablet_service.cc:1511] Processing CreateTablet for tablet a26c6a2ffb8746a3941a56c7259412e3 (DEFAULT_TABLE table=default.Table_225ad80b8318432b88a729064c88e8ec [id=a3cca003025745cebf8e7d403d0778cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.533351 9506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a26c6a2ffb8746a3941a56c7259412e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.543550 10393 raft_consensus.cc:2749] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:24.544260 10393 raft_consensus.cc:2749] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:24.544392 9507 tablet_service.cc:1511] Processing CreateTablet for tablet 2f6b3e969760442283c3191e062c1839 (DEFAULT_TABLE table=default.Table_5bc32a610edd49ab9641c3726f53f5c6 [id=be5c761518674e7782bfff73366b1675]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.545452 9507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f6b3e969760442283c3191e062c1839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.547665 9536 raft_consensus.cc:1275] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: Refusing update from remote peer d4051d2bf0bf410eb8acd107560df888: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260504 13:56:24.547750 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 1501455a0af348e38746d100987afdd0 (table default.Table_9b8d10cfdddc499a86ba289a87dbe689 [id=1fcd0497578846cb9bbd71e501606912]) successfully deleted
I20260504 13:56:24.548003 9536 pending_rounds.cc:85] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00: Aborting all ops after (but not including) 0
I20260504 13:56:24.548209 9536 pending_rounds.cc:107] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260504 13:56:24.548420 9536 raft_consensus.cc:2889] T acd5f7d3f7c545d18fe19b64e4bc1de6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 13:56:24.578023 10497 consensus_queue.cc:1048] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:24.581175 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "604f1ba5f71041a7877daff8972a994a" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:24.582082 9681 raft_consensus.cc:2468] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:24.590322 9932 tablet_bootstrap.cc:492] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:24.596280 9511 tablet_service.cc:1511] Processing CreateTablet for tablet c47b6e3b4d0c45c28384f7aae6b749de (DEFAULT_TABLE table=default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.597592 9511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c47b6e3b4d0c45c28384f7aae6b749de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.597831 10486 tablet.cc:1724] T b13303d2a14c4187b82da9f25ce9753e P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:24.600579 10497 raft_consensus.cc:493] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:24.601032 10497 raft_consensus.cc:515] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.607331 10497 leader_election.cc:290] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:24.610495 9768 leader_election.cc:343] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: c34630d55f4a4e19b33ddf4c13995a5b
I20260504 13:56:24.613000 10496 ts_tablet_manager.cc:1916] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:24.618049 9338 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_1230367a2b2e46f6bf1e4c8628f1eff1" } new_table_name: "default.Table_c4e8170ca1324d54b2bd104e88eaeb6a" modify_external_catalogs: true
I20260504 13:56:24.622913 9335 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_0e2a1bfda0a945dc86c7e8e9bd28fa49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:24.624390 9335 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e2a1bfda0a945dc86c7e8e9bd28fa49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:24.626919 9341 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_bff621bd8f674f7fa3cdb3de52bb8386" } new_table_name: "default.Table_ae10fcacbc05403bb0331bd9c5f8cda4" modify_external_catalogs: true
I20260504 13:56:24.628094 9932 ts_tablet_manager.cc:1403] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.377s user 0.008s sys 0.011s
I20260504 13:56:24.640384 9505 tablet_service.cc:1511] Processing CreateTablet for tablet 2a629c2a69084b6b98984c8d374eeb1e (DEFAULT_TABLE table=default.Table_a5fa0a0bead94aa3b02d6175fefde171 [id=e9fbd43d30ae4d1dbeea534025125b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.641422 9505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a629c2a69084b6b98984c8d374eeb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.643205 9944 ts_tablet_manager.cc:1929] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:24.643637 9944 log.cc:1199] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/48b193e958b4414aad29dd2bcee1a61b
I20260504 13:56:24.644506 9944 ts_tablet_manager.cc:1950] T 48b193e958b4414aad29dd2bcee1a61b P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:24.664503 9509 tablet_service.cc:1511] Processing CreateTablet for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (DEFAULT_TABLE table=default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.665596 9509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.672231 9512 tablet_service.cc:1511] Processing CreateTablet for tablet 4a1a77ad56634e068401a2bfbc977761 (DEFAULT_TABLE table=default.Table_9bc232a8e4dd4df1a36b35d09dd50d0a [id=66faab34fb324213a874ea6fae3c96a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.673293 9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a1a77ad56634e068401a2bfbc977761. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:24.675508 9328 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c47b6e3b4d0c45c28384f7aae6b749de consensus state has no leader
I20260504 13:56:24.676846 9931 tablet_bootstrap.cc:492] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:24.677409 9931 ts_tablet_manager.cc:1403] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.491s user 0.008s sys 0.002s
I20260504 13:56:24.679700 9931 raft_consensus.cc:359] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:24.680318 9931 raft_consensus.cc:385] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:24.680574 9931 raft_consensus.cc:740] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:24.681141 9931 consensus_queue.cc:260] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:24.689929 9931 ts_tablet_manager.cc:1434] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260504 13:56:24.695364 9931 tablet_bootstrap.cc:492] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:24.695860 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 108023us
I20260504 13:56:24.703500 10486 tablet.cc:1724] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260504 13:56:24.704813 9767 leader_election.cc:343] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Not found: Tablet not found: c34630d55f4a4e19b33ddf4c13995a5b
I20260504 13:56:24.705880 10562 tablet.cc:1724] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:24.705582 9767 leader_election.cc:304] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:24.707712 10578 raft_consensus.cc:2749] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:24.708676 10578 consensus_queue.cc:1048] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:24.721315 10496 ts_tablet_manager.cc:1929] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:24.725975 10414 raft_consensus.cc:493] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:24.726478 10414 raft_consensus.cc:515] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.730064 10496 log.cc:1199] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/baf07a970a7c4a35b3c849b06d6c88d0
I20260504 13:56:24.730826 10496 ts_tablet_manager.cc:1950] T baf07a970a7c4a35b3c849b06d6c88d0 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:24.730861 10414 leader_election.cc:290] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:24.722723 9931 tablet_bootstrap.cc:654] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:24.729511 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3c045379734134b49e94340679c11d" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:24.734141 9474 leader_election.cc:343] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:24.735773 10013 ts_tablet_manager.cc:1434] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.341s user 0.005s sys 0.010s
I20260504 13:56:24.736640 10013 tablet_bootstrap.cc:492] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:24.747540 9330 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 85124.9us
I20260504 13:56:24.749374 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]) successfully deleted
W20260504 13:56:24.756168 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 604f1ba5f71041a7877daff8972a994a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 604f1ba5f71041a7877daff8972a994a already in progress: creating tablet
I20260504 13:56:24.756944 10496 tablet_replica.cc:333] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:24.757745 10496 raft_consensus.cc:2243] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [term 2 LEADER]: Raft consensus shutting down.
I20260504 13:56:24.758692 10496 raft_consensus.cc:2272] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:24.762142 9931 tablet_bootstrap.cc:492] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:24.762552 9931 ts_tablet_manager.cc:1403] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.072s user 0.007s sys 0.002s
I20260504 13:56:24.764748 9931 raft_consensus.cc:359] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:24.765329 9931 raft_consensus.cc:385] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:24.765569 9931 raft_consensus.cc:740] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:24.776680 10496 ts_tablet_manager.cc:1916] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:24.783416 9931 consensus_queue.cc:260] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:24.777150 9336 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_7f9a1296bc424f7aa21625c670da0dd0" } modify_external_catalogs: true
I20260504 13:56:24.786832 9336 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_7f9a1296bc424f7aa21625c670da0dd0" } modify_external_catalogs: true
I20260504 13:56:24.787873 9931 ts_tablet_manager.cc:1434] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.025s user 0.005s sys 0.002s
I20260504 13:56:24.790769 9931 tablet_bootstrap.cc:492] T dd3c045379734134b49e94340679c11d P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:24.795969 9931 tablet_bootstrap.cc:654] T dd3c045379734134b49e94340679c11d P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:24.799322 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 70caac2e256d419dbc2e70cb5dfffe97 (table default.Table_c5584202645f4ef78e4f6e00aa2bf2c1 [id=2665806fb81f4eda8981fede847cbfcf]) successfully deleted
I20260504 13:56:24.814899 10496 ts_tablet_manager.cc:1929] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20260504 13:56:24.815276 10496 log.cc:1199] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/c34877a81d504f46a2f680c5d5ddb4f4
I20260504 13:56:24.816005 10496 ts_tablet_manager.cc:1950] T c34877a81d504f46a2f680c5d5ddb4f4 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:24.816792 10382 raft_consensus.cc:493] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:24.817216 10382 raft_consensus.cc:515] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:24.819005 10382 leader_election.cc:290] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:24.820807 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030b35c3409143108c34ab1f0ee1b96d" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:24.842247 9474 consensus_peers.cc:597] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: baf07a970a7c4a35b3c849b06d6c88d0. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:24.844372 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet c34877a81d504f46a2f680c5d5ddb4f4 (table default.Table_c6a898ba23b74b54bb4684e970b44742 [id=92ea89acf1fc427690ee0a61cbbb1647]) successfully deleted
I20260504 13:56:24.845312 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030b35c3409143108c34ab1f0ee1b96d" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:24.852192 9932 raft_consensus.cc:359] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.853149 9932 raft_consensus.cc:385] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:24.853541 9932 raft_consensus.cc:740] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:24.854298 9932 consensus_queue.cc:260] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:24.857652 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3c045379734134b49e94340679c11d" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:24.858189 9684 raft_consensus.cc:2468] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:24.858655 9656 tablet_service.cc:1511] Processing CreateTablet for tablet ba4584755aae47e0b0f14c6c55474d18 (DEFAULT_TABLE table=default.Table_2d5a582adb634895a55193dd861f3b60 [id=30f349197d564e8da82855fec17b4cd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.859448 9657 tablet_service.cc:1511] Processing CreateTablet for tablet f3e86027e3e74bc196e9a9ffc9fdf8a5 (DEFAULT_TABLE table=default.Table_780e641042db4fd5aeb8af8b1aa08c72 [id=6cb26719d5fe45ccbdef18571c902228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.860066 9656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba4584755aae47e0b0f14c6c55474d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.859910 9658 tablet_service.cc:1511] Processing CreateTablet for tablet de6714ee86d64112baf13dd1b2b87582 (DEFAULT_TABLE table=default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.861012 9658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de6714ee86d64112baf13dd1b2b87582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.861049 9665 tablet_service.cc:1511] Processing CreateTablet for tablet a26c6a2ffb8746a3941a56c7259412e3 (DEFAULT_TABLE table=default.Table_225ad80b8318432b88a729064c88e8ec [id=a3cca003025745cebf8e7d403d0778cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.862396 9657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e86027e3e74bc196e9a9ffc9fdf8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.864146 9663 tablet_service.cc:1511] Processing CreateTablet for tablet 2f6b3e969760442283c3191e062c1839 (DEFAULT_TABLE table=default.Table_5bc32a610edd49ab9641c3726f53f5c6 [id=be5c761518674e7782bfff73366b1675]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.864705 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a26c6a2ffb8746a3941a56c7259412e3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:24.872788 9768 consensus_peers.cc:597] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50e129980cd84f6ca22cbf2addb06230. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:24.873597 9768 leader_election.cc:304] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:24.874511 10586 raft_consensus.cc:2749] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:24.960942 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f6b3e969760442283c3191e062c1839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.968770 9660 tablet_service.cc:1511] Processing CreateTablet for tablet 2a629c2a69084b6b98984c8d374eeb1e (DEFAULT_TABLE table=default.Table_a5fa0a0bead94aa3b02d6175fefde171 [id=e9fbd43d30ae4d1dbeea534025125b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.969779 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a629c2a69084b6b98984c8d374eeb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.969491 9659 tablet_service.cc:1511] Processing CreateTablet for tablet c47b6e3b4d0c45c28384f7aae6b749de (DEFAULT_TABLE table=default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.970592 9659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c47b6e3b4d0c45c28384f7aae6b749de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.971254 9666 tablet_service.cc:1511] Processing CreateTablet for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (DEFAULT_TABLE table=default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.972215 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:24.972419 9662 tablet_service.cc:1511] Processing CreateTablet for tablet 4a1a77ad56634e068401a2bfbc977761 (DEFAULT_TABLE table=default.Table_9bc232a8e4dd4df1a36b35d09dd50d0a [id=66faab34fb324213a874ea6fae3c96a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:24.973456 9662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a1a77ad56634e068401a2bfbc977761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:25.018791 9833 raft_consensus.cc:3060] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:25.019850 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 150) took 2688 ms. Trace:
I20260504 13:56:25.020218 9944 rpcz_store.cc:276] 0504 13:56:22.331013 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:22.331142 (+ 129us) service_pool.cc:224] Handling call
0504 13:56:24.157318 (+1826176us) ts_tablet_manager.cc:1151] Deleting tablet 48b193e958b4414aad29dd2bcee1a61b
0504 13:56:24.157361 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:24.302746 (+145385us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:24.642941 (+340195us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:25.019760 (+376819us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67856,"tablet-delete.queue_time_us":1733286}
I20260504 13:56:25.025692 9944 tablet_replica.cc:333] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:25.035145 9944 raft_consensus.cc:2243] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:25.036689 9944 raft_consensus.cc:2272] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:25.038998 9944 ts_tablet_manager.cc:1916] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:25.040412 9932 ts_tablet_manager.cc:1434] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.412s user 0.007s sys 0.001s
I20260504 13:56:25.043071 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 48b193e958b4414aad29dd2bcee1a61b (table default.Table_2c39b14515bb46f6a0b0f1252d8e273d [id=e9e56499fd2243919308927b3614ae8e]) successfully deleted
I20260504 13:56:25.074082 9339 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35768 (request call id 246) took 1090 ms. Trace:
I20260504 13:56:25.074404 9339 rpcz_store.cc:276] 0504 13:56:23.984060 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:24.202198 (+218138us) service_pool.cc:224] Handling call
0504 13:56:24.216192 (+ 13994us) catalog_manager.cc:2101] Acquired catalog manager lock
0504 13:56:24.301697 (+ 85505us) catalog_manager.cc:2147] Created new table and tablet info
0504 13:56:25.073436 (+771739us) catalog_manager.cc:2207] Wrote table and tablets to system table
0504 13:56:25.073749 (+ 313us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0504 13:56:25.074028 (+ 279us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:24.362839 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:24.662335 (+299496us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:24.662361 (+ 26us) write_op.cc:435] Acquired schema lock
0504 13:56:24.662372 (+ 11us) tablet.cc:662] Decoding operations
0504 13:56:24.662662 (+ 290us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:24.662698 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:24.662720 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:24.662896 (+ 176us) tablet.cc:701] Row locks acquired
0504 13:56:24.662909 (+ 13us) write_op.cc:260] PREPARE: finished
0504 13:56:24.663019 (+ 110us) write_op.cc:270] Start()
0504 13:56:24.663112 (+ 93us) write_op.cc:276] Timestamp: P: 1777902984662963 usec, L: 0
0504 13:56:24.663140 (+ 28us) op_driver.cc:348] REPLICATION: starting
0504 13:56:24.755665 (+ 92525us) log.cc:844] Serialized 433 byte log entry
0504 13:56:25.071599 (+315934us) op_driver.cc:464] REPLICATION: finished
0504 13:56:25.071818 (+ 219us) write_op.cc:301] APPLY: starting
0504 13:56:25.071876 (+ 58us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:25.071992 (+ 116us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:25.072080 (+ 88us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:25.072473 (+ 393us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:25.072604 (+ 131us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0504 13:56:25.072649 (+ 45us) write_op.cc:312] APPLY: finished
0504 13:56:25.072841 (+ 192us) log.cc:844] Serialized 29 byte log entry
0504 13:56:25.072993 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:25.073055 (+ 62us) write_op.cc:454] Released schema lock
0504 13:56:25.073086 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":78,"apply.run_cpu_time_us":1460,"apply.run_wall_time_us":1461,"num_ops":2,"prepare.queue_time_us":60028,"prepare.run_cpu_time_us":2038,"prepare.run_wall_time_us":401276,"raft.queue_time_us":478,"raft.run_cpu_time_us":535,"raft.run_wall_time_us":11379,"replication_time_us":408364}]]}
I20260504 13:56:25.095042 9473 leader_election.cc:304] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:25.098543 10563 raft_consensus.cc:2804] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:25.098856 10563 raft_consensus.cc:493] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:25.099246 10563 raft_consensus.cc:3060] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:25.102530 9944 ts_tablet_manager.cc:1929] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:25.103111 9944 log.cc:1199] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/030b35c3409143108c34ab1f0ee1b96d
I20260504 13:56:25.121798 10563 raft_consensus.cc:515] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:25.130184 9944 ts_tablet_manager.cc:1950] T 030b35c3409143108c34ab1f0ee1b96d P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:25.135456 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "604f1ba5f71041a7877daff8972a994a" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:25.136008 9829 raft_consensus.cc:3060] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:25.151741 9829 raft_consensus.cc:2468] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:25.157326 9474 leader_election.cc:304] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:25.158136 10431 raft_consensus.cc:2804] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:25.159300 10431 raft_consensus.cc:697] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:25.160205 10431 consensus_queue.cc:237] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:25.163596 10577 log.cc:927] Time spent T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6: Append to log took a long time: real 0.503s user 0.000s sys 0.001s
I20260504 13:56:25.164618 9932 tablet_bootstrap.cc:492] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:25.190676 10582 log.cc:927] Time spent T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260504 13:56:25.194821 9931 tablet_bootstrap.cc:492] T dd3c045379734134b49e94340679c11d P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:25.195338 9931 ts_tablet_manager.cc:1403] T dd3c045379734134b49e94340679c11d P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.405s user 0.060s sys 0.034s
I20260504 13:56:25.198167 9931 raft_consensus.cc:359] T dd3c045379734134b49e94340679c11d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.209513 9932 tablet_bootstrap.cc:654] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:25.223897 10566 raft_consensus.cc:493] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4051d2bf0bf410eb8acd107560df888)
I20260504 13:56:25.224411 10566 raft_consensus.cc:515] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:25.226388 10566 leader_election.cc:290] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
W20260504 13:56:25.229295 9474 leader_election.cc:343] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Not found: Tablet not found: 1cf2b59d13da40a2b2160ebdc0b2eb80
W20260504 13:56:25.230692 9474 consensus_peers.cc:597] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:25.188285 10391 ts_tablet_manager.cc:1929] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:25.192200 10563 leader_election.cc:290] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:25.255859 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 153) took 2255 ms. Trace:
I20260504 13:56:25.256198 9944 rpcz_store.cc:276] 0504 13:56:23.000400 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:23.000546 (+ 146us) service_pool.cc:224] Handling call
0504 13:56:25.021402 (+2020856us) ts_tablet_manager.cc:1151] Deleting tablet 030b35c3409143108c34ab1f0ee1b96d
0504 13:56:25.025592 (+ 4190us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:25.050503 (+ 24911us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:25.102402 (+ 51899us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:25.255799 (+153397us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8164,"tablet-delete.queue_time_us":1928060}
I20260504 13:56:25.257314 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 030b35c3409143108c34ab1f0ee1b96d (table default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]) successfully deleted
I20260504 13:56:25.258174 9944 tablet_replica.cc:333] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:25.258836 9944 raft_consensus.cc:2243] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:25.259286 9944 raft_consensus.cc:2272] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:25.234218 9508 consensus_meta.cc:287] T de6714ee86d64112baf13dd1b2b87582 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent flushing consensus metadata: real 0.664s user 0.007s sys 0.000s
I20260504 13:56:25.265018 9499 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:52636 (request call id 24) took 1142 ms. Trace:
I20260504 13:56:25.249972 10013 tablet_bootstrap.cc:654] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:25.264027 9512 consensus_meta.cc:287] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent flushing consensus metadata: real 0.582s user 0.007s sys 0.000s
I20260504 13:56:25.234700 10391 log.cc:1199] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/afad9329cf7745ba8168b2a93f998295
I20260504 13:56:25.268098 10391 ts_tablet_manager.cc:1950] T afad9329cf7745ba8168b2a93f998295 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:25.268882 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 146) took 3998 ms. Trace:
I20260504 13:56:25.273881 9931 raft_consensus.cc:385] T dd3c045379734134b49e94340679c11d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:25.274209 9931 raft_consensus.cc:740] T dd3c045379734134b49e94340679c11d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:25.281049 9337 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet fe7ab3b577b74a9c9a40cb15c94f8aa1 (table default.Table_dee1335eff89438e9c68835a4c258031 [id=1714df2b31ff4865b6b357500d121a26]). Expected version 1 got 0
I20260504 13:56:25.288069 9337 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet b13303d2a14c4187b82da9f25ce9753e (table default.Table_72e3a254561b4cae927e6b6bcacd87b6 [id=f227576f105c4b1fbd459b3638ce684b]). Expected version 2 got 0
I20260504 13:56:25.274889 9931 consensus_queue.cc:260] T dd3c045379734134b49e94340679c11d P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.295042 9931 ts_tablet_manager.cc:1434] T dd3c045379734134b49e94340679c11d P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.099s user 0.006s sys 0.000s
I20260504 13:56:25.297719 9931 tablet_bootstrap.cc:492] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:25.302839 9931 tablet_bootstrap.cc:654] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:25.303382 9509 consensus_meta.cc:287] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent flushing consensus metadata: real 0.619s user 0.002s sys 0.006s
I20260504 13:56:25.337307 9932 tablet_bootstrap.cc:492] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:25.337785 9932 ts_tablet_manager.cc:1403] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.187s user 0.002s sys 0.007s
I20260504 13:56:25.340665 9932 raft_consensus.cc:359] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.341260 9932 raft_consensus.cc:385] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:25.331007 10391 rpcz_store.cc:276] 0504 13:56:21.270740 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.282570 (+ 11830us) service_pool.cc:224] Handling call
0504 13:56:24.361242 (+3078672us) ts_tablet_manager.cc:1151] Deleting tablet afad9329cf7745ba8168b2a93f998295
0504 13:56:24.361283 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:24.510010 (+148727us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:25.188167 (+678157us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:25.268818 (+ 80651us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3059705}
I20260504 13:56:25.341495 9932 raft_consensus.cc:740] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:25.342180 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 137) took 4075 ms. Trace:
I20260504 13:56:25.342111 9932 consensus_queue.cc:260] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.352581 10391 rpcz_store.cc:276] 0504 13:56:21.266835 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.419435 (+152600us) service_pool.cc:224] Handling call
0504 13:56:25.341899 (+3922464us) ts_tablet_manager.cc:1151] Deleting tablet a43cdb767fd14ea09b4db4d2881b69aa
0504 13:56:25.341943 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:25.342125 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3921509}
W20260504 13:56:25.352033 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 242337us
I20260504 13:56:25.373085 9473 leader_election.cc:304] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:25.376523 9944 ts_tablet_manager.cc:1916] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:25.379587 10414 raft_consensus.cc:2804] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:25.379896 10414 raft_consensus.cc:493] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:25.380174 10414 raft_consensus.cc:3060] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:25.386803 10414 raft_consensus.cc:515] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.388510 10414 leader_election.cc:290] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:25.311983 9505 consensus_meta.cc:287] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent flushing consensus metadata: real 0.659s user 0.009s sys 0.000s
I20260504 13:56:25.265324 9499 rpcz_store.cc:276] 0504 13:56:24.122691 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:24.123207 (+ 516us) service_pool.cc:224] Handling call
0504 13:56:24.468789 (+345582us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b54000f2b54. stack: 00007f591ed51c83 00007f591ed51b80 00007f591e8c2574 000000000050e094 000000000050e029 00007f5925da9951 00007f5925e3a2d8 00007f5926e7cd9d 00007f5925601104 00007f5925601092 00007f5925600fc1 00007f5925600f3c 00007f59256001f2 00007f5923b33bcc 00007f5923b33006 00007f5923b329af
0504 13:56:24.730106 (+261317us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b5400101654. stack: 00007f591ed51c83 00007f591ed51b80 00007f591e8c2574 000000000050e094 000000000050e029 00007f5925da9951 00007f5925e3a2d8 00007f5926e7cd9d 00007f5925601104 00007f5925601092 00007f5925600fc1 00007f5925600f3c 00007f59256001f2 00007f5923b33bcc 00007f5923b33006 00007f5923b329af
0504 13:56:25.264947 (+534841us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1030166784}
W20260504 13:56:25.309504 9338 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92480654807d495b879576b0195871bd has no consensus state
I20260504 13:56:25.393559 9338 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35768 (request call id 248) took 1032 ms. Trace:
I20260504 13:56:25.394874 10571 raft_consensus.cc:493] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 13:56:25.317754 9341 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dffddd4b89984c54bc26dcb4a9488525 has no consensus state
I20260504 13:56:25.395525 9341 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35768 (request call id 250) took 1022 ms. Trace:
I20260504 13:56:25.395334 10571 raft_consensus.cc:515] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.397176 10571 leader_election.cc:290] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:25.398350 9932 ts_tablet_manager.cc:1434] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.060s user 0.006s sys 0.001s
I20260504 13:56:25.395789 9341 rpcz_store.cc:276] 0504 13:56:24.372633 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:24.372737 (+ 104us) service_pool.cc:224] Handling call
0504 13:56:24.626828 (+254091us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b0fb6 00007fc27b2c6b24 00007fc27b2c6ab2 00007fc27b2c69e1 00007fc27b2c695c 00007fc27b2c5c12 00007fc27958bbcc 00007fc27958b006
0504 13:56:24.627453 (+ 625us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:24.629091 (+ 1638us) catalog_manager.cc:3725] Apply alter schema
0504 13:56:24.629569 (+ 478us) catalog_manager.cc:3757] Acquired catalog manager lock
0504 13:56:24.629746 (+ 177us) catalog_manager.cc:3915] Updating metadata on disk
0504 13:56:25.317241 (+687495us) catalog_manager.cc:3946] Committing alterations to in-memory state
0504 13:56:25.395472 (+ 78231us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:24.943608 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:24.944069 (+ 461us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:24.944093 (+ 24us) write_op.cc:435] Acquired schema lock
0504 13:56:24.944104 (+ 11us) tablet.cc:662] Decoding operations
0504 13:56:24.944342 (+ 238us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:24.944373 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:24.944394 (+ 21us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:24.944524 (+ 130us) tablet.cc:701] Row locks acquired
0504 13:56:24.944535 (+ 11us) write_op.cc:260] PREPARE: finished
0504 13:56:24.944660 (+ 125us) write_op.cc:270] Start()
0504 13:56:24.944755 (+ 95us) write_op.cc:276] Timestamp: P: 1777902984944612 usec, L: 0
0504 13:56:24.944776 (+ 21us) op_driver.cc:348] REPLICATION: starting
0504 13:56:25.271658 (+326882us) log.cc:844] Serialized 405 byte log entry
0504 13:56:25.273064 (+ 1406us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b801951 00007fc27b894208 00007fc27bd069f1 00007fc27bd061c8 00007fc27bd0ae61 00007fc27bd0ae19 00007fc27bd0ada9 00007fc27bd0ad71 00007fc27bd09f9d 00000000005b3bc4 00000000005b39f9
0504 13:56:25.315691 (+ 42627us) op_driver.cc:464] REPLICATION: finished
0504 13:56:25.315889 (+ 198us) write_op.cc:301] APPLY: starting
0504 13:56:25.315942 (+ 53us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:25.316022 (+ 80us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:25.316041 (+ 19us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:25.316214 (+ 173us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:25.316372 (+ 158us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:25.316432 (+ 60us) write_op.cc:312] APPLY: finished
0504 13:56:25.316619 (+ 187us) log.cc:844] Serialized 23 byte log entry
0504 13:56:25.316755 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:25.316815 (+ 60us) write_op.cc:454] Released schema lock
0504 13:56:25.316852 (+ 37us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":566783360,"child_traces":[["op",{"apply.queue_time_us":70,"apply.run_cpu_time_us":1155,"apply.run_wall_time_us":1154,"num_ops":1,"prepare.queue_time_us":312841,"prepare.run_cpu_time_us":4442,"prepare.run_wall_time_us":329645,"raft.queue_time_us":3495,"raft.run_cpu_time_us":879,"raft.run_wall_time_us":1382,"replication_time_us":370827,"spinlock_wait_cycles":752042496}]]}
I20260504 13:56:25.365572 9336 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1
I20260504 13:56:25.400688 9336 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35768 (request call id 249) took 1036 ms. Trace:
I20260504 13:56:25.401461 9662 tablet_service.cc:1558] Processing DeleteTablet for tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:24 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:25.375507 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet afad9329cf7745ba8168b2a93f998295 (table default.Table_cdee25533fe54da0a74b421714b0d7c4 [id=81df524d85774544807ba1902be1aae1]) successfully deleted
I20260504 13:56:25.402879 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d63f665166466dba1e7e36c5e19506" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:25.404011 9618 leader_election.cc:343] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:25.393843 9338 rpcz_store.cc:276] 0504 13:56:24.361251 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:24.388251 (+ 27000us) service_pool.cc:224] Handling call
0504 13:56:24.617965 (+229714us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b0fb6 00007fc27b2c6b24 00007fc27b2c6ab2 00007fc27b2c69e1 00007fc27b2c695c 00007fc27b2c5c12 00007fc27958bbcc 00007fc27958b006
0504 13:56:24.618626 (+ 661us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:24.620126 (+ 1500us) catalog_manager.cc:3725] Apply alter schema
0504 13:56:24.620727 (+ 601us) catalog_manager.cc:3757] Acquired catalog manager lock
0504 13:56:24.620930 (+ 203us) catalog_manager.cc:3915] Updating metadata on disk
0504 13:56:25.308988 (+688058us) catalog_manager.cc:3946] Committing alterations to in-memory state
0504 13:56:25.393495 (+ 84507us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:24.764230 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:24.787675 (+ 23445us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:24.787699 (+ 24us) write_op.cc:435] Acquired schema lock
0504 13:56:24.787711 (+ 12us) tablet.cc:662] Decoding operations
0504 13:56:24.787957 (+ 246us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:24.787991 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:24.788012 (+ 21us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:24.792729 (+ 4717us) tablet.cc:701] Row locks acquired
0504 13:56:24.792744 (+ 15us) write_op.cc:260] PREPARE: finished
0504 13:56:24.792846 (+ 102us) write_op.cc:270] Start()
0504 13:56:24.792921 (+ 75us) write_op.cc:276] Timestamp: P: 1777902984792783 usec, L: 0
0504 13:56:24.792936 (+ 15us) op_driver.cc:348] REPLICATION: starting
0504 13:56:24.793435 (+ 499us) log.cc:844] Serialized 405 byte log entry
0504 13:56:25.307446 (+514011us) op_driver.cc:464] REPLICATION: finished
0504 13:56:25.307661 (+ 215us) write_op.cc:301] APPLY: starting
0504 13:56:25.307721 (+ 60us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:25.307811 (+ 90us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:25.307904 (+ 93us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:25.308060 (+ 156us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:25.308202 (+ 142us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:25.308251 (+ 49us) write_op.cc:312] APPLY: finished
0504 13:56:25.308427 (+ 176us) log.cc:844] Serialized 23 byte log entry
0504 13:56:25.308568 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:25.308627 (+ 59us) write_op.cc:454] Released schema lock
0504 13:56:25.308664 (+ 37us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":526710016,"child_traces":[["op",{"apply.queue_time_us":81,"apply.run_cpu_time_us":1177,"apply.run_wall_time_us":1176,"num_ops":1,"prepare.queue_time_us":142350,"prepare.run_cpu_time_us":2030,"prepare.run_wall_time_us":179252,"raft.queue_time_us":107,"raft.run_cpu_time_us":737,"raft.run_wall_time_us":12861,"replication_time_us":514414}]]}
I20260504 13:56:25.405476 9338 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_5bfac12ab2f84c06b1f992848d8f5041" } modify_external_catalogs: true
I20260504 13:56:25.405938 9338 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_5bfac12ab2f84c06b1f992848d8f5041" } modify_external_catalogs: true
W20260504 13:56:25.407749 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet a43cdb767fd14ea09b4db4d2881b69aa because the tablet was not found. No further retry: Not found: Tablet not found: a43cdb767fd14ea09b4db4d2881b69aa
I20260504 13:56:25.408154 9341 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_1611aa9f4dba40e09dae4433e2d20154"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:25.409684 9341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1611aa9f4dba40e09dae4433e2d20154 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:25.401003 9336 rpcz_store.cc:276] 0504 13:56:24.364072 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:24.420144 (+ 56072us) service_pool.cc:224] Handling call
0504 13:56:24.777044 (+356900us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b0c46 00007fc27b2c2024 00007fc27b2c1fb2 00007fc27b2c1ee1 00007fc27b2c1e5c 00007fc27b2c1112 00007fc27958bbcc 00007fc27958b006
0504 13:56:24.791926 (+ 14882us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:24.823966 (+ 32040us) catalog_manager.cc:2815] Modifying in-memory table state
0504 13:56:24.824114 (+ 148us) catalog_manager.cc:2824] Locking tablets
0504 13:56:24.825024 (+ 910us) catalog_manager.cc:2838] Removing table and tablets from system table
0504 13:56:25.365237 (+540213us) catalog_manager.cc:2856] Removing table from by-name map
0504 13:56:25.365405 (+ 168us) catalog_manager.cc:2872] Committing in-memory state
0504 13:56:25.365477 (+ 72us) catalog_manager.cc:2876] Aborting table tasks
0504 13:56:25.400637 (+ 35160us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:25.286297 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:25.286670 (+ 373us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:25.286693 (+ 23us) write_op.cc:435] Acquired schema lock
0504 13:56:25.286706 (+ 13us) tablet.cc:662] Decoding operations
0504 13:56:25.287116 (+ 410us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:25.287157 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:25.287203 (+ 46us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:25.287493 (+ 290us) tablet.cc:701] Row locks acquired
0504 13:56:25.287511 (+ 18us) write_op.cc:260] PREPARE: finished
0504 13:56:25.287627 (+ 116us) write_op.cc:270] Start()
0504 13:56:25.287697 (+ 70us) write_op.cc:276] Timestamp: P: 1777902985287560 usec, L: 0
0504 13:56:25.287711 (+ 14us) op_driver.cc:348] REPLICATION: starting
0504 13:56:25.288107 (+ 396us) log.cc:844] Serialized 731 byte log entry
0504 13:56:25.361476 (+ 73369us) op_driver.cc:464] REPLICATION: finished
0504 13:56:25.363978 (+ 2502us) write_op.cc:301] APPLY: starting
0504 13:56:25.364049 (+ 71us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:25.364154 (+ 105us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:25.364235 (+ 81us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:25.364492 (+ 257us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:25.364621 (+ 129us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:25.364664 (+ 43us) write_op.cc:312] APPLY: finished
0504 13:56:25.364831 (+ 167us) log.cc:844] Serialized 29 byte log entry
0504 13:56:25.364963 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:25.365031 (+ 68us) write_op.cc:454] Released schema lock
0504 13:56:25.365059 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":817893760,"child_traces":[["op",{"apply.queue_time_us":2348,"apply.run_cpu_time_us":1291,"apply.run_wall_time_us":1960,"num_ops":2,"prepare.queue_time_us":459538,"prepare.run_cpu_time_us":2009,"prepare.run_wall_time_us":19284,"raft.queue_time_us":63,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":183,"replication_time_us":73679}]]}
I20260504 13:56:25.414767 9336 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_f00a5967b1b7422b830dcf11db2acbd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:25.417207 10571 raft_consensus.cc:493] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:25.417615 10571 raft_consensus.cc:515] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:25.420323 9336 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f00a5967b1b7422b830dcf11db2acbd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:25.422698 10571 leader_election.cc:290] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:25.423497 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8702ce3ff1a845f3b70012855cd17efd" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:25.424667 9618 leader_election.cc:343] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:25.435034 9932 tablet_bootstrap.cc:492] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:25.438068 9341 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35768 (request call id 252) took 1029 ms. Trace:
I20260504 13:56:25.442019 9932 tablet_bootstrap.cc:654] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:25.407663 9337 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 0be795f04dd54cf19869c937b91d0ddf (table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]). Expected version 1 got 0
I20260504 13:56:25.378683 10013 tablet_bootstrap.cc:492] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:25.455592 10013 ts_tablet_manager.cc:1403] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.719s user 0.005s sys 0.006s
W20260504 13:56:25.456420 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 77003.4us
I20260504 13:56:25.372495 10391 tablet_replica.cc:333] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:25.483422 10602 tablet.cc:1724] T 0be795f04dd54cf19869c937b91d0ddf P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:25.438537 9341 rpcz_store.cc:276] 0504 13:56:24.408545 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:25.399123 (+990578us) service_pool.cc:224] Handling call
0504 13:56:25.410129 (+ 11006us) catalog_manager.cc:2101] Acquired catalog manager lock
0504 13:56:25.411576 (+ 1447us) catalog_manager.cc:2147] Created new table and tablet info
0504 13:56:25.437320 (+ 25744us) catalog_manager.cc:2207] Wrote table and tablets to system table
0504 13:56:25.437661 (+ 341us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0504 13:56:25.437987 (+ 326us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:25.412839 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:25.413346 (+ 507us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:25.413396 (+ 50us) write_op.cc:435] Acquired schema lock
0504 13:56:25.413416 (+ 20us) tablet.cc:662] Decoding operations
0504 13:56:25.413649 (+ 233us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:25.413685 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:25.413706 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:25.413864 (+ 158us) tablet.cc:701] Row locks acquired
0504 13:56:25.413876 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:25.413972 (+ 96us) write_op.cc:270] Start()
0504 13:56:25.414045 (+ 73us) write_op.cc:276] Timestamp: P: 1777902985413904 usec, L: 0
0504 13:56:25.414059 (+ 14us) op_driver.cc:348] REPLICATION: starting
0504 13:56:25.414489 (+ 430us) log.cc:844] Serialized 433 byte log entry
0504 13:56:25.422713 (+ 8224us) op_driver.cc:464] REPLICATION: finished
0504 13:56:25.423101 (+ 388us) write_op.cc:301] APPLY: starting
0504 13:56:25.423218 (+ 117us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:25.423352 (+ 134us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:25.423469 (+ 117us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:25.423933 (+ 464us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:25.424149 (+ 216us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0504 13:56:25.424216 (+ 67us) write_op.cc:312] APPLY: finished
0504 13:56:25.424448 (+ 232us) log.cc:844] Serialized 29 byte log entry
0504 13:56:25.424609 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:25.424696 (+ 87us) write_op.cc:454] Released schema lock
0504 13:56:25.424741 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:25.425006 (+ 265us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:25.425064 (+ 58us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":217,"apply.run_cpu_time_us":1851,"apply.run_wall_time_us":1863,"num_ops":2,"prepare.queue_time_us":91,"prepare.run_cpu_time_us":1902,"prepare.run_wall_time_us":1903,"raft.queue_time_us":1041,"raft.run_cpu_time_us":624,"raft.run_wall_time_us":7361,"replication_time_us":8555}]]}
W20260504 13:56:25.356904 10591 log.cc:927] Time spent T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888: Append to log took a long time: real 0.050s user 0.000s sys 0.001s
I20260504 13:56:25.395865 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3c045379734134b49e94340679c11d" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:25.498415 9833 raft_consensus.cc:3060] T dd3c045379734134b49e94340679c11d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:25.402372 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d63f665166466dba1e7e36c5e19506" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:25.501758 9620 leader_election.cc:343] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:25.502135 9620 leader_election.cc:304] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:25.504202 10573 raft_consensus.cc:2749] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:25.403931 9806 tablet_service.cc:1558] Processing DeleteTablet for tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:24 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:25.426292 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8702ce3ff1a845f3b70012855cd17efd" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:25.490979 10601 tablet.cc:1724] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260504 13:56:25.507613 9620 leader_election.cc:343] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:25.507967 9620 leader_election.cc:304] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:25.509518 9833 raft_consensus.cc:2468] T dd3c045379734134b49e94340679c11d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:25.510273 10571 raft_consensus.cc:2749] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:25.515926 9341 master_service.cc:954] ReplaceTablet: received request to replace tablet 7c9030802900499a90c4dbaf5c6936a4 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:25.497421 9336 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_09255abca80f41dc895d81ccd89b9c7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:25.524255 9474 leader_election.cc:304] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:25.536708 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "604f1ba5f71041a7877daff8972a994a" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:25.537485 9681 raft_consensus.cc:3060] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:25.551892 9328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260504 13:56:15.938203 10203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:56:15.924326 10208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:56:25.558072 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd3c045379734134b49e94340679c11d" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:25.558564 9684 raft_consensus.cc:3060] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:25.549747 10431 raft_consensus.cc:2804] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:25.563112 10431 raft_consensus.cc:697] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:25.563850 10431 consensus_queue.cc:237] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.569788 9684 raft_consensus.cc:2468] T dd3c045379734134b49e94340679c11d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:25.576711 9505 tablet_service.cc:1558] Processing DeleteTablet for tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:24 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:25.570178 9932 tablet_bootstrap.cc:492] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:25.578644 9932 ts_tablet_manager.cc:1403] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.144s user 0.009s sys 0.000s
I20260504 13:56:25.580859 9932 raft_consensus.cc:359] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:25.581410 9932 raft_consensus.cc:385] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:25.581632 9932 raft_consensus.cc:740] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:25.583092 10086 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260504 13:56:25.583781 10207 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 9679 milliseconds
I20260504 13:56:25.584292 10391 raft_consensus.cc:2243] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:25.585253 10391 raft_consensus.cc:2272] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:25.585583 10086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260504 13:56:25.587469 10391 ts_tablet_manager.cc:1916] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:25.592222 10013 raft_consensus.cc:359] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.592923 10013 raft_consensus.cc:385] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:25.593183 10013 raft_consensus.cc:740] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
W20260504 13:56:25.518846 9336 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09255abca80f41dc895d81ccd89b9c7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:25.591455 9944 ts_tablet_manager.cc:1929] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:25.594134 9944 log.cc:1199] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/930078a10afa4dd987ad021cf9609514
I20260504 13:56:25.594921 9944 ts_tablet_manager.cc:1950] T 930078a10afa4dd987ad021cf9609514 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:25.597666 9932 consensus_queue.cc:260] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:25.599018 10086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260504 13:56:25.599665 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1 already in progress: creating tablet
I20260504 13:56:25.600852 9931 tablet_bootstrap.cc:492] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:25.601243 9931 ts_tablet_manager.cc:1403] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.305s user 0.009s sys 0.000s
I20260504 13:56:25.610984 10013 consensus_queue.cc:260] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.614758 9931 raft_consensus.cc:359] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.616945 9931 raft_consensus.cc:385] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:25.617197 9931 raft_consensus.cc:740] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:25.617614 10566 raft_consensus.cc:493] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:25.617772 9931 consensus_queue.cc:260] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.617998 10566 raft_consensus.cc:515] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.621052 10566 leader_election.cc:290] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:25.623849 10086 hybrid_clock.cc:648] HybridClock initialized: now 1777902985623779 us; error 51 us; skew 500 ppm
I20260504 13:56:25.625106 10086 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 13:56:25.630244 9931 ts_tablet_manager.cc:1434] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20260504 13:56:25.634949 9931 tablet_bootstrap.cc:492] T 7ed3ee9a13e84594905db47aa33040c2 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:25.638624 9768 consensus_peers.cc:597] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afad9329cf7745ba8168b2a93f998295. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:25.639986 10607 tablet.cc:1724] T 0be795f04dd54cf19869c937b91d0ddf P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:25.644946 9931 tablet_bootstrap.cc:654] T 7ed3ee9a13e84594905db47aa33040c2 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:25.652341 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ed3ee9a13e84594905db47aa33040c2" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:25.656858 9932 ts_tablet_manager.cc:1434] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.078s user 0.005s sys 0.000s
I20260504 13:56:25.658978 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 156) took 2377 ms. Trace:
I20260504 13:56:25.659400 9944 rpcz_store.cc:276] 0504 13:56:23.281234 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:23.281360 (+ 126us) service_pool.cc:224] Handling call
0504 13:56:25.257952 (+1976592us) ts_tablet_manager.cc:1151] Deleting tablet 930078a10afa4dd987ad021cf9609514
0504 13:56:25.258024 (+ 72us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:25.466996 (+208972us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:25.591299 (+124303us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:25.658904 (+ 67605us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1976071}
I20260504 13:56:25.660647 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 930078a10afa4dd987ad021cf9609514 (table default.Table_0d2ce6c9ff35497b82dd58da3f913ecb [id=5a0c5354346746c186e36fe23406442c]) successfully deleted
I20260504 13:56:25.666895 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ed3ee9a13e84594905db47aa33040c2" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:25.667415 9686 raft_consensus.cc:2468] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:25.668088 10602 tablet.cc:1724] T fe7ab3b577b74a9c9a40cb15c94f8aa1 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:25.672045 10600 raft_consensus.cc:493] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:25.672505 10600 raft_consensus.cc:515] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.674191 10600 leader_election.cc:290] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:25.675539 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1cf52a9e58744a99dae1958d2c1f867" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:25.676162 9684 raft_consensus.cc:2393] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:25.683633 9932 tablet_bootstrap.cc:492] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:25.688974 9330 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet dffddd4b89984c54bc26dcb4a9488525 (table default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet dffddd4b89984c54bc26dcb4a9488525 has no consensus state
W20260504 13:56:25.655634 9474 leader_election.cc:343] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:25.692075 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1cf52a9e58744a99dae1958d2c1f867" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:25.692731 9767 consensus_peers.cc:597] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afad9329cf7745ba8168b2a93f998295. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:25.693451 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 159) took 1993 ms. Trace:
I20260504 13:56:25.693534 9767 leader_election.cc:304] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:25.694439 10600 raft_consensus.cc:3060] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:25.694804 9944 rpcz_store.cc:276] 0504 13:56:23.700406 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:23.839265 (+138859us) service_pool.cc:224] Handling call
0504 13:56:25.693090 (+1853825us) ts_tablet_manager.cc:1151] Deleting tablet 8e4e583afc804212bece2f727411cc14
0504 13:56:25.693148 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:25.693391 (+ 243us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1853148}
I20260504 13:56:25.695574 9931 tablet_bootstrap.cc:492] T 7ed3ee9a13e84594905db47aa33040c2 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:25.695964 9931 ts_tablet_manager.cc:1403] T 7ed3ee9a13e84594905db47aa33040c2 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.000s
W20260504 13:56:25.698809 9328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 146942us
I20260504 13:56:25.702984 10600 raft_consensus.cc:2749] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:25.663004 10013 ts_tablet_manager.cc:1434] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.207s user 0.006s sys 0.000s
I20260504 13:56:25.704213 9944 tablet_replica.cc:333] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:25.704787 10013 tablet_bootstrap.cc:492] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:25.704747 9329 catalog_manager.cc:4982] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 8e4e583afc804212bece2f727411cc14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e4e583afc804212bece2f727411cc14 already in progress: creating tablet
I20260504 13:56:25.705895 9931 raft_consensus.cc:359] T 7ed3ee9a13e84594905db47aa33040c2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.706488 9931 raft_consensus.cc:385] T 7ed3ee9a13e84594905db47aa33040c2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:25.706735 9931 raft_consensus.cc:740] T 7ed3ee9a13e84594905db47aa33040c2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:25.707362 9931 consensus_queue.cc:260] T 7ed3ee9a13e84594905db47aa33040c2 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.708381 9944 raft_consensus.cc:2243] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260504 13:56:25.716583 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 48992.5us
I20260504 13:56:25.717232 9944 raft_consensus.cc:2272] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:25.717278 10431 consensus_queue.cc:1048] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:25.719489 9944 ts_tablet_manager.cc:1916] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260504 13:56:25.720504 9473 leader_election.cc:343] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 1cf2b59d13da40a2b2160ebdc0b2eb80
I20260504 13:56:25.720850 9473 leader_election.cc:304] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:25.721873 10566 raft_consensus.cc:2749] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:25.724031 9931 ts_tablet_manager.cc:1434] T 7ed3ee9a13e84594905db47aa33040c2 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20260504 13:56:25.725375 9932 tablet_bootstrap.cc:654] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:25.730683 9681 raft_consensus.cc:2468] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:25.742749 10013 tablet_bootstrap.cc:654] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:25.742991 9931 tablet_bootstrap.cc:492] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:25.748394 9931 tablet_bootstrap.cc:654] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:25.757478 9473 leader_election.cc:304] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:25.758980 10431 raft_consensus.cc:2804] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:25.759372 10431 raft_consensus.cc:493] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:25.759775 10431 raft_consensus.cc:3060] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:25.789731 9932 tablet_bootstrap.cc:492] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:25.790200 9932 ts_tablet_manager.cc:1403] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.107s user 0.011s sys 0.002s
I20260504 13:56:25.792320 9932 raft_consensus.cc:359] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.792884 9932 raft_consensus.cc:385] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:25.793113 9932 raft_consensus.cc:740] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:25.793671 9932 consensus_queue.cc:260] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.797839 9944 ts_tablet_manager.cc:1929] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:25.798197 9944 log.cc:1199] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/9ed4f5ea230a413a8dcdbb661ce173b6
I20260504 13:56:25.798880 9944 ts_tablet_manager.cc:1950] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:25.800872 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 164) took 1887 ms. Trace:
I20260504 13:56:25.801174 9944 rpcz_store.cc:276] 0504 13:56:23.913255 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:23.913374 (+ 119us) service_pool.cc:224] Handling call
0504 13:56:25.704071 (+1790697us) ts_tablet_manager.cc:1151] Deleting tablet 9ed4f5ea230a413a8dcdbb661ce173b6
0504 13:56:25.704117 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:25.753893 (+ 49776us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:25.771173 (+ 17280us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:25.800819 (+ 29646us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1790121}
I20260504 13:56:25.802182 9944 tablet_replica.cc:333] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:25.802312 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]) successfully deleted
I20260504 13:56:25.802866 9944 raft_consensus.cc:2243] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:25.807590 10086 webserver.cc:492] Webserver started at http://127.8.238.253:44077/ using document root <none> and password file <none>
I20260504 13:56:25.809048 10086 fs_manager.cc:362] Metadata directory not provided
I20260504 13:56:25.809361 10086 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 13:56:25.814376 9944 raft_consensus.cc:2272] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:25.814886 10571 raft_consensus.cc:493] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:25.815335 10571 raft_consensus.cc:515] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.817181 10571 leader_election.cc:290] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:25.820730 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1aba9a2a58b4e6b9e87f8ec80fcb6f1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:25.821900 9618 leader_election.cc:343] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:25.822451 10431 raft_consensus.cc:515] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.823486 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1aba9a2a58b4e6b9e87f8ec80fcb6f1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:25.824406 10431 leader_election.cc:290] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:25.824714 9620 leader_election.cc:343] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:25.824936 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ed3ee9a13e84594905db47aa33040c2" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:25.825119 9944 ts_tablet_manager.cc:1916] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:25.825038 9620 leader_election.cc:304] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:25.825443 9833 raft_consensus.cc:3060] T 7ed3ee9a13e84594905db47aa33040c2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:25.830531 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ed3ee9a13e84594905db47aa33040c2" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:25.831003 9684 raft_consensus.cc:3060] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:25.832932 9833 raft_consensus.cc:2468] T 7ed3ee9a13e84594905db47aa33040c2 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:25.833727 9474 leader_election.cc:304] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:25.834358 10611 raft_consensus.cc:2804] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:25.834733 10611 raft_consensus.cc:697] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:25.832373 9932 ts_tablet_manager.cc:1434] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.042s user 0.007s sys 0.000s
I20260504 13:56:25.835474 10611 consensus_queue.cc:237] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.835338 10573 raft_consensus.cc:2749] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:25.836634 9932 tablet_bootstrap.cc:492] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:25.844738 10013 tablet_bootstrap.cc:492] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:25.845122 10013 ts_tablet_manager.cc:1403] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.141s user 0.008s sys 0.000s
I20260504 13:56:25.847682 10013 raft_consensus.cc:359] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.848392 10013 raft_consensus.cc:385] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:25.848671 10013 raft_consensus.cc:740] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:25.857352 9684 raft_consensus.cc:2468] T 7ed3ee9a13e84594905db47aa33040c2 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:25.875888 9944 ts_tablet_manager.cc:1929] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:25.876245 9944 log.cc:1199] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/604f1ba5f71041a7877daff8972a994a
I20260504 13:56:25.876897 9944 ts_tablet_manager.cc:1950] T 604f1ba5f71041a7877daff8972a994a P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:25.878804 9932 tablet_bootstrap.cc:654] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:25.878860 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 168) took 1621 ms. Trace:
I20260504 13:56:25.879240 9944 rpcz_store.cc:276] 0504 13:56:24.257172 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:24.257369 (+ 197us) service_pool.cc:224] Handling call
0504 13:56:25.802013 (+1544644us) ts_tablet_manager.cc:1151] Deleting tablet 604f1ba5f71041a7877daff8972a994a
0504 13:56:25.802077 (+ 64us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:25.845974 (+ 43897us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:25.875770 (+ 29796us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:25.878805 (+ 3035us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5,"tablet-delete.queue_time_us":1532620}
I20260504 13:56:25.880293 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 604f1ba5f71041a7877daff8972a994a (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]) successfully deleted
I20260504 13:56:25.892971 10013 consensus_queue.cc:260] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.894821 10013 ts_tablet_manager.cc:1434] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20260504 13:56:25.899533 9944 tablet_replica.cc:333] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:25.902205 9944 raft_consensus.cc:2243] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:25.902685 9944 raft_consensus.cc:2272] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:25.892314 10391 ts_tablet_manager.cc:1929] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:25.907858 10391 log.cc:1199] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/65052a5c28c147a385144317e21eea41
I20260504 13:56:25.908679 10391 ts_tablet_manager.cc:1950] T 65052a5c28c147a385144317e21eea41 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:25.909008 9931 tablet_bootstrap.cc:492] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:25.909404 9931 ts_tablet_manager.cc:1403] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.167s user 0.021s sys 0.012s
I20260504 13:56:25.910871 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 136) took 4644 ms. Trace:
I20260504 13:56:25.911703 9931 raft_consensus.cc:359] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.911545 10391 rpcz_store.cc:276] 0504 13:56:21.266507 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.419555 (+153048us) service_pool.cc:224] Handling call
0504 13:56:25.372348 (+3952793us) ts_tablet_manager.cc:1151] Deleting tablet 65052a5c28c147a385144317e21eea41
0504 13:56:25.372389 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:25.737283 (+364894us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:25.892194 (+154911us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:25.910808 (+ 18614us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60527,"tablet-delete.queue_time_us":3949882}
I20260504 13:56:25.912304 9931 raft_consensus.cc:385] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:25.912547 9931 raft_consensus.cc:740] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:25.912858 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 65052a5c28c147a385144317e21eea41 (table default.Table_038807318bcc4110b77de9582f6c42f5 [id=c19c0f3f26934ad6bbf2d4202cdaa542]) successfully deleted
I20260504 13:56:25.913306 9931 consensus_queue.cc:260] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.914006 9944 ts_tablet_manager.cc:1916] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:25.929478 9931 ts_tablet_manager.cc:1434] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20260504 13:56:25.931906 10086 fs_manager.cc:714] Time spent opening directory manager: real 0.089s user 0.001s sys 0.010s
I20260504 13:56:25.947834 9931 tablet_bootstrap.cc:492] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:25.954345 9931 tablet_bootstrap.cc:654] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:25.913841 10391 tablet_replica.cc:333] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:25.967653 9944 ts_tablet_manager.cc:1929] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:25.967799 10623 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 13:56:25.967984 9944 log.cc:1199] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/baf07a970a7c4a35b3c849b06d6c88d0
I20260504 13:56:25.955442 10013 tablet_bootstrap.cc:492] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:25.968639 9944 ts_tablet_manager.cc:1950] T baf07a970a7c4a35b3c849b06d6c88d0 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:25.971683 10086 fs_manager.cc:730] Time spent opening block manager: real 0.037s user 0.004s sys 0.003s
W20260504 13:56:25.975210 9473 consensus_peers.cc:597] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 604f1ba5f71041a7877daff8972a994a. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:25.976914 9932 tablet_bootstrap.cc:492] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:25.977319 9932 ts_tablet_manager.cc:1403] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.141s user 0.004s sys 0.004s
I20260504 13:56:25.972113 10086 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/data,/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/wal
uuid: "6caedf407d13459e8fd477e8b60b7dc7"
format_stamp: "Formatted at 2026-05-04 13:55:56 on dist-test-slave-f9wt"
I20260504 13:56:25.978515 10086 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 13:56:25.979732 9932 raft_consensus.cc:359] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.980289 9932 raft_consensus.cc:385] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:25.980511 9932 raft_consensus.cc:740] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:25.981124 9932 consensus_queue.cc:260] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:25.975813 10391 raft_consensus.cc:2243] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:25.986917 10391 raft_consensus.cc:2272] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:25.987906 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 167) took 1731 ms. Trace:
I20260504 13:56:25.988232 9944 rpcz_store.cc:276] 0504 13:56:24.256925 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:24.262064 (+ 5139us) service_pool.cc:224] Handling call
0504 13:56:25.899376 (+1637312us) ts_tablet_manager.cc:1151] Deleting tablet baf07a970a7c4a35b3c849b06d6c88d0
0504 13:56:25.899432 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:25.935564 (+ 36132us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:25.967536 (+ 31972us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:25.987851 (+ 20315us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1629757}
I20260504 13:56:25.988837 10391 ts_tablet_manager.cc:1916] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:25.989352 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 171) took 1691 ms. Trace:
I20260504 13:56:25.989630 9944 rpcz_store.cc:276] 0504 13:56:24.298007 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:24.298132 (+ 125us) service_pool.cc:224] Handling call
0504 13:56:25.989061 (+1690929us) ts_tablet_manager.cc:1151] Deleting tablet c34630d55f4a4e19b33ddf4c13995a5b
0504 13:56:25.989133 (+ 72us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:25.989309 (+ 176us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1690448}
I20260504 13:56:25.991703 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 170) took 1694 ms. Trace:
I20260504 13:56:25.992002 9944 rpcz_store.cc:276] 0504 13:56:24.296838 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:24.296969 (+ 131us) service_pool.cc:224] Handling call
0504 13:56:25.990356 (+1693387us) ts_tablet_manager.cc:1151] Deleting tablet c34877a81d504f46a2f680c5d5ddb4f4
0504 13:56:25.991492 (+ 1136us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:25.991660 (+ 168us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1690065}
I20260504 13:56:25.992719 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]) successfully deleted
I20260504 13:56:25.993206 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 172) took 1686 ms. Trace:
W20260504 13:56:25.993361 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet c34630d55f4a4e19b33ddf4c13995a5b because the tablet was not found. No further retry: Not found: Tablet not found: c34630d55f4a4e19b33ddf4c13995a5b
I20260504 13:56:25.993462 9944 rpcz_store.cc:276] 0504 13:56:24.307251 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:24.307340 (+ 89us) service_pool.cc:224] Handling call
0504 13:56:25.992959 (+1685619us) ts_tablet_manager.cc:1151] Deleting tablet bd8cd1caaeb249e3bafee7f1f0d4df2f
0504 13:56:25.993002 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:25.993156 (+ 154us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1685157}
W20260504 13:56:25.993875 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet c34877a81d504f46a2f680c5d5ddb4f4 because the tablet was not found. No further retry: Not found: Tablet not found: c34877a81d504f46a2f680c5d5ddb4f4
W20260504 13:56:25.994447 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f because the tablet was not found. No further retry: Not found: Tablet not found: bd8cd1caaeb249e3bafee7f1f0d4df2f
I20260504 13:56:25.994547 9944 tablet_replica.cc:333] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:25.995260 9944 raft_consensus.cc:2243] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:25.995694 9944 raft_consensus.cc:2272] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:25.997499 9944 ts_tablet_manager.cc:1916] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:25.998584 10013 tablet_bootstrap.cc:654] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.007450 9932 ts_tablet_manager.cc:1434] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260504 13:56:26.008165 9931 tablet_bootstrap.cc:492] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:26.008576 9931 ts_tablet_manager.cc:1403] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.004s
I20260504 13:56:26.010560 9932 tablet_bootstrap.cc:492] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:26.010848 9931 raft_consensus.cc:359] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.011440 9931 raft_consensus.cc:385] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.011662 9931 raft_consensus.cc:740] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.019421 9931 consensus_queue.cc:260] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.029641 9931 ts_tablet_manager.cc:1434] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260504 13:56:26.033978 9944 ts_tablet_manager.cc:1929] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:26.034312 9944 log.cc:1199] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/1501455a0af348e38746d100987afdd0
I20260504 13:56:26.034682 9931 tablet_bootstrap.cc:492] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:26.034977 9944 ts_tablet_manager.cc:1950] T 1501455a0af348e38746d100987afdd0 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:26.034840 9932 tablet_bootstrap.cc:654] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.036916 9944 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40388 (request call id 165) took 1884 ms. Trace:
I20260504 13:56:26.037204 9944 rpcz_store.cc:276] 0504 13:56:24.152379 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:24.524566 (+372187us) service_pool.cc:224] Handling call
0504 13:56:25.994419 (+1469853us) ts_tablet_manager.cc:1151] Deleting tablet 1501455a0af348e38746d100987afdd0
0504 13:56:25.994453 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:26.024925 (+ 30472us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:26.033878 (+ 8953us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:26.036866 (+ 2988us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1468493}
I20260504 13:56:26.038332 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 1501455a0af348e38746d100987afdd0 (table default.Table_9b8d10cfdddc499a86ba289a87dbe689 [id=1fcd0497578846cb9bbd71e501606912]) successfully deleted
I20260504 13:56:26.044677 9944 tablet_replica.cc:333] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:26.045370 9944 raft_consensus.cc:2243] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:26.045759 9944 raft_consensus.cc:2272] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:26.046271 10571 raft_consensus.cc:493] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:26.046665 10571 raft_consensus.cc:515] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:26.048779 9944 ts_tablet_manager.cc:1916] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:26.049183 10571 leader_election.cc:290] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:26.050398 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c9030802900499a90c4dbaf5c6936a4" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:26.051967 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c9030802900499a90c4dbaf5c6936a4" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:26.053612 9620 leader_election.cc:343] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:26.054234 9618 leader_election.cc:343] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:26.054606 9618 leader_election.cc:304] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:26.059327 10571 raft_consensus.cc:2749] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:26.060736 9931 tablet_bootstrap.cc:654] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.073774 10013 tablet_bootstrap.cc:492] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:26.074218 10013 ts_tablet_manager.cc:1403] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.119s user 0.010s sys 0.000s
I20260504 13:56:26.075026 9944 ts_tablet_manager.cc:1929] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:26.075380 9944 log.cc:1199] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/a1aba9a2a58b4e6b9e87f8ec80fcb6f1
I20260504 13:56:26.075930 9944 ts_tablet_manager.cc:1950] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:26.076924 10013 raft_consensus.cc:359] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.078056 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1 (table default.Table_7f9a1296bc424f7aa21625c670da0dd0 [id=a62d0dfdbe7741248a4d5d6614c712eb]) successfully deleted
I20260504 13:56:26.077653 10013 raft_consensus.cc:385] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.078797 10013 raft_consensus.cc:740] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.079501 10013 consensus_queue.cc:260] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.081883 10013 ts_tablet_manager.cc:1434] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260504 13:56:26.085119 9932 tablet_bootstrap.cc:492] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:26.085556 9932 ts_tablet_manager.cc:1403] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.075s user 0.013s sys 0.000s
I20260504 13:56:26.088214 9932 raft_consensus.cc:359] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.088094 10611 raft_consensus.cc:493] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:26.089182 10600 raft_consensus.cc:493] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:26.089104 10611 raft_consensus.cc:515] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.089586 10600 raft_consensus.cc:515] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.090826 10611 leader_election.cc:290] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.091267 10600 leader_election.cc:290] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.092900 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8db7c4732fa048f9950248f45d144c57" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:26.094059 9767 leader_election.cc:343] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:26.094134 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3112c3c3629412e9b76407c7cba8f92" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:26.095402 9474 leader_election.cc:343] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:26.092579 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3112c3c3629412e9b76407c7cba8f92" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:26.097424 9684 raft_consensus.cc:2468] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:26.098488 9473 leader_election.cc:304] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:26.093063 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8db7c4732fa048f9950248f45d144c57" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:26.099506 9681 raft_consensus.cc:2468] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:26.100648 9768 leader_election.cc:304] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:26.102593 9931 tablet_bootstrap.cc:492] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:26.102943 9931 ts_tablet_manager.cc:1403] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.068s user 0.009s sys 0.000s
I20260504 13:56:26.103142 10586 raft_consensus.cc:2804] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:26.103487 10586 raft_consensus.cc:493] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:26.103760 10586 raft_consensus.cc:3060] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.105116 9931 raft_consensus.cc:359] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.105712 9931 raft_consensus.cc:385] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.106005 9931 raft_consensus.cc:740] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.106550 9931 consensus_queue.cc:260] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.106956 10566 raft_consensus.cc:2804] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:26.107345 10566 raft_consensus.cc:493] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:26.107678 10566 raft_consensus.cc:3060] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.117184 9931 ts_tablet_manager.cc:1434] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.014s user 0.007s sys 0.001s
I20260504 13:56:26.118007 9931 tablet_bootstrap.cc:492] T c3112c3c3629412e9b76407c7cba8f92 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:26.118312 9932 raft_consensus.cc:385] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.096617 10013 tablet_bootstrap.cc:492] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:26.122269 10586 raft_consensus.cc:515] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.125416 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8db7c4732fa048f9950248f45d144c57" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
W20260504 13:56:26.126721 9767 leader_election.cc:343] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:26.122826 9931 tablet_bootstrap.cc:654] T c3112c3c3629412e9b76407c7cba8f92 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.127261 9932 raft_consensus.cc:740] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.117965 10566 raft_consensus.cc:515] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.128464 9932 consensus_queue.cc:260] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.131049 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3112c3c3629412e9b76407c7cba8f92" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:26.132367 9474 leader_election.cc:343] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:26.141322 9931 tablet_bootstrap.cc:492] T c3112c3c3629412e9b76407c7cba8f92 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:26.141708 9931 ts_tablet_manager.cc:1403] T c3112c3c3629412e9b76407c7cba8f92 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.001s
I20260504 13:56:26.142110 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8db7c4732fa048f9950248f45d144c57" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:26.142638 9681 raft_consensus.cc:3060] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.144065 9931 raft_consensus.cc:359] T c3112c3c3629412e9b76407c7cba8f92 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.144789 9931 raft_consensus.cc:385] T c3112c3c3629412e9b76407c7cba8f92 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.145041 9931 raft_consensus.cc:740] T c3112c3c3629412e9b76407c7cba8f92 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.145922 10586 leader_election.cc:290] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.151650 9681 raft_consensus.cc:2468] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:26.152709 9768 leader_election.cc:304] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:26.154922 9932 ts_tablet_manager.cc:1434] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.069s user 0.005s sys 0.000s
I20260504 13:56:26.145641 9931 consensus_queue.cc:260] T c3112c3c3629412e9b76407c7cba8f92 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.155483 10629 raft_consensus.cc:2804] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:26.155895 10629 raft_consensus.cc:697] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:26.156651 10629 consensus_queue.cc:237] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.163525 9931 ts_tablet_manager.cc:1434] T c3112c3c3629412e9b76407c7cba8f92 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20260504 13:56:26.163899 10566 leader_election.cc:290] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.164335 9931 tablet_bootstrap.cc:492] T 7cb6927537704cc4b17617cbdaa21ac1 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:26.164386 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3112c3c3629412e9b76407c7cba8f92" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:26.164835 9684 raft_consensus.cc:3060] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.171237 9684 raft_consensus.cc:2468] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:26.173890 10013 tablet_bootstrap.cc:654] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.177500 9473 leader_election.cc:304] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:26.178589 10566 raft_consensus.cc:2804] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:26.181584 10566 raft_consensus.cc:697] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:26.182417 10566 consensus_queue.cc:237] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.174734 9932 tablet_bootstrap.cc:492] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:26.188215 9931 tablet_bootstrap.cc:654] T 7cb6927537704cc4b17617cbdaa21ac1 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:26.197758 9473 consensus_peers.cc:597] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ed4f5ea230a413a8dcdbb661ce173b6. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:26.215389 9931 tablet_bootstrap.cc:492] T 7cb6927537704cc4b17617cbdaa21ac1 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:26.215752 9931 ts_tablet_manager.cc:1403] T 7cb6927537704cc4b17617cbdaa21ac1 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.000s
I20260504 13:56:26.218060 9931 raft_consensus.cc:359] T 7cb6927537704cc4b17617cbdaa21ac1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.218614 9931 raft_consensus.cc:385] T 7cb6927537704cc4b17617cbdaa21ac1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.218852 9931 raft_consensus.cc:740] T 7cb6927537704cc4b17617cbdaa21ac1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.220983 10391 ts_tablet_manager.cc:1929] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260504 13:56:26.221383 10391 log.cc:1199] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/1cf2b59d13da40a2b2160ebdc0b2eb80
I20260504 13:56:26.222110 10391 ts_tablet_manager.cc:1950] T 1cf2b59d13da40a2b2160ebdc0b2eb80 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:26.224182 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 148) took 4352 ms. Trace:
I20260504 13:56:26.231992 9931 consensus_queue.cc:260] T 7cb6927537704cc4b17617cbdaa21ac1 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.233840 9931 ts_tablet_manager.cc:1434] T 7cb6927537704cc4b17617cbdaa21ac1 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20260504 13:56:26.237828 9931 tablet_bootstrap.cc:492] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:26.238695 10571 raft_consensus.cc:493] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:26.239120 10571 raft_consensus.cc:515] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.240943 10571 leader_election.cc:290] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:26.242871 9931 tablet_bootstrap.cc:654] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.250068 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaa40eb4b1c425a9edfb7eaba953b65" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:26.251394 9620 leader_election.cc:343] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:26.271034 9932 tablet_bootstrap.cc:654] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.241932 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaa40eb4b1c425a9edfb7eaba953b65" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:26.284479 10391 rpcz_store.cc:276] 0504 13:56:21.871531 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:21.888972 (+ 17441us) service_pool.cc:224] Handling call
0504 13:56:25.913708 (+4024736us) ts_tablet_manager.cc:1151] Deleting tablet 1cf2b59d13da40a2b2160ebdc0b2eb80
0504 13:56:25.913746 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:26.058051 (+144305us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:26.220873 (+162822us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:26.224113 (+ 3240us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4022240}
I20260504 13:56:26.285010 9931 tablet_bootstrap.cc:492] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
W20260504 13:56:26.285585 9618 leader_election.cc:343] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:26.285658 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 1cf2b59d13da40a2b2160ebdc0b2eb80 (table default.Table_0eae04160cf241df95cc53da6f4dfacf [id=3822050e2a9d4a9db8b0e36430fd0b7f]) successfully deleted
I20260504 13:56:26.285943 9931 ts_tablet_manager.cc:1403] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.000s
I20260504 13:56:26.285976 9618 leader_election.cc:304] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:26.287025 10571 raft_consensus.cc:2749] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:26.285995 10391 tablet_replica.cc:333] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:26.285343 10013 tablet_bootstrap.cc:492] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:26.288518 10013 ts_tablet_manager.cc:1403] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.192s user 0.032s sys 0.019s
I20260504 13:56:26.288393 9931 raft_consensus.cc:359] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.288875 10391 raft_consensus.cc:2243] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:26.288993 9931 raft_consensus.cc:385] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.289240 9931 raft_consensus.cc:740] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.289922 10391 raft_consensus.cc:2272] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:26.289796 9931 consensus_queue.cc:260] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.293054 10013 raft_consensus.cc:359] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.293634 10013 raft_consensus.cc:385] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.293860 10013 raft_consensus.cc:740] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.294476 9932 tablet_bootstrap.cc:492] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:26.294416 10013 consensus_queue.cc:260] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.294942 9932 ts_tablet_manager.cc:1403] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.120s user 0.008s sys 0.003s
I20260504 13:56:26.302454 10013 ts_tablet_manager.cc:1434] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.014s user 0.003s sys 0.002s
I20260504 13:56:26.303254 10013 tablet_bootstrap.cc:492] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:26.313570 9931 ts_tablet_manager.cc:1434] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20260504 13:56:26.314437 9931 tablet_bootstrap.cc:492] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:26.322965 10013 tablet_bootstrap.cc:654] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.337651 10391 ts_tablet_manager.cc:1916] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:26.298377 9932 raft_consensus.cc:359] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.345324 9932 raft_consensus.cc:385] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.345661 9932 raft_consensus.cc:740] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.346357 9932 consensus_queue.cc:260] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.358408 9932 ts_tablet_manager.cc:1434] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.063s user 0.005s sys 0.000s
I20260504 13:56:26.373529 10611 consensus_queue.cc:1048] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:26.376236 10013 tablet_bootstrap.cc:492] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:26.376648 10013 ts_tablet_manager.cc:1403] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.074s user 0.009s sys 0.004s
I20260504 13:56:26.378858 10013 raft_consensus.cc:359] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.383898 10013 raft_consensus.cc:385] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.384291 10013 raft_consensus.cc:740] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.384985 10013 consensus_queue.cc:260] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.386317 10013 ts_tablet_manager.cc:1434] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.009s user 0.004s sys 0.001s
I20260504 13:56:26.388154 10013 tablet_bootstrap.cc:492] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:26.392688 9932 tablet_bootstrap.cc:492] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:26.398191 9932 tablet_bootstrap.cc:654] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.400817 10013 tablet_bootstrap.cc:654] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.429471 9931 tablet_bootstrap.cc:654] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.437768 10431 consensus_queue.cc:1048] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:26.439440 10571 raft_consensus.cc:493] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:26.440004 10571 raft_consensus.cc:515] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.442216 10571 leader_election.cc:290] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:26.443754 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9077fb9bb1544a2cbeb96f0d11c400db" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:26.445772 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9077fb9bb1544a2cbeb96f0d11c400db" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:26.447268 9620 leader_election.cc:343] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:26.447906 9618 leader_election.cc:343] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:26.448225 9618 leader_election.cc:304] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:26.452112 10571 raft_consensus.cc:2749] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:26.474839 10566 consensus_queue.cc:1048] T dd3c045379734134b49e94340679c11d P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:26.493497 10013 tablet_bootstrap.cc:492] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:26.493897 10013 ts_tablet_manager.cc:1403] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.106s user 0.018s sys 0.016s
I20260504 13:56:26.496165 10013 raft_consensus.cc:359] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.496754 10013 raft_consensus.cc:385] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.496994 10013 raft_consensus.cc:740] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.503265 10013 consensus_queue.cc:260] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.506246 10013 ts_tablet_manager.cc:1434] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260504 13:56:26.508744 9932 tablet_bootstrap.cc:492] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:26.508982 9931 tablet_bootstrap.cc:492] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:26.509331 9931 ts_tablet_manager.cc:1403] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.195s user 0.008s sys 0.001s
I20260504 13:56:26.509290 9932 ts_tablet_manager.cc:1403] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.117s user 0.047s sys 0.058s
I20260504 13:56:26.512046 9932 raft_consensus.cc:359] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.512720 9932 raft_consensus.cc:385] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.512953 9932 raft_consensus.cc:740] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.514554 10013 tablet_bootstrap.cc:492] T c8989cdde4ca4e349079596e33723b2c P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:26.513490 9932 consensus_queue.cc:260] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.519501 9932 ts_tablet_manager.cc:1434] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260504 13:56:26.520282 9932 tablet_bootstrap.cc:492] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:26.522933 10013 tablet_bootstrap.cc:654] T c8989cdde4ca4e349079596e33723b2c P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.535615 9931 raft_consensus.cc:359] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.536206 9931 raft_consensus.cc:385] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.536463 9932 tablet_bootstrap.cc:654] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.536507 9931 raft_consensus.cc:740] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.537076 9931 consensus_queue.cc:260] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.538914 9931 ts_tablet_manager.cc:1434] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.029s user 0.001s sys 0.002s
I20260504 13:56:26.549418 10391 ts_tablet_manager.cc:1929] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:26.549847 10391 log.cc:1199] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/3c43ff1d759a40d68344f25f066416ac
I20260504 13:56:26.550568 10391 ts_tablet_manager.cc:1950] T 3c43ff1d759a40d68344f25f066416ac P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:26.555779 9931 tablet_bootstrap.cc:492] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:26.561242 9931 tablet_bootstrap.cc:654] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.568591 10586 raft_consensus.cc:493] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:26.568985 10586 raft_consensus.cc:515] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.570760 10586 leader_election.cc:290] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.572418 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5254b1061e2b424487f5d309baaee6b6" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:26.572971 9681 raft_consensus.cc:2468] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:26.574097 9768 leader_election.cc:304] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:26.575317 10629 raft_consensus.cc:2804] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:26.575670 10629 raft_consensus.cc:493] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:26.575912 10654 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:26.576041 10629 raft_consensus.cc:3060] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.576473 10654 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:26.581543 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 302 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:26.586180 10571 raft_consensus.cc:493] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:26.586586 10571 raft_consensus.cc:515] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.587342 10431 raft_consensus.cc:493] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:26.587744 10431 raft_consensus.cc:515] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.588285 10571 leader_election.cc:290] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:26.596822 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5254b1061e2b424487f5d309baaee6b6" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:26.597422 9538 raft_consensus.cc:2468] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:26.598338 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3404ba04d3ec4a44a5e625536b941b58" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:26.599870 9618 leader_election.cc:343] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:26.602284 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3404ba04d3ec4a44a5e625536b941b58" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:26.604390 9620 leader_election.cc:343] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:26.604861 9620 leader_election.cc:304] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:26.605983 10571 raft_consensus.cc:2749] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:26.608558 10431 leader_election.cc:290] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.579103 10654 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:26.610050 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5254b1061e2b424487f5d309baaee6b6" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:26.611166 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5254b1061e2b424487f5d309baaee6b6" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:26.611668 9684 raft_consensus.cc:2468] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:26.612665 9473 leader_election.cc:304] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:26.613600 10431 raft_consensus.cc:2804] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:26.613991 10431 raft_consensus.cc:493] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:26.614557 10431 raft_consensus.cc:3060] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.621232 9932 tablet_bootstrap.cc:492] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:26.621656 9932 ts_tablet_manager.cc:1403] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.102s user 0.010s sys 0.006s
I20260504 13:56:26.625399 10600 raft_consensus.cc:493] T c3112c3c3629412e9b76407c7cba8f92 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:26.625957 10600 raft_consensus.cc:515] T c3112c3c3629412e9b76407c7cba8f92 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.629371 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3112c3c3629412e9b76407c7cba8f92" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:26.630048 9684 raft_consensus.cc:2393] T c3112c3c3629412e9b76407c7cba8f92 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:26.630573 10600 leader_election.cc:290] T c3112c3c3629412e9b76407c7cba8f92 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.635237 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3112c3c3629412e9b76407c7cba8f92" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:26.635805 10611 consensus_queue.cc:1048] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:26.638820 9932 raft_consensus.cc:359] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:26.639436 9932 raft_consensus.cc:385] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.639683 9932 raft_consensus.cc:740] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.638950 9767 leader_election.cc:304] T c3112c3c3629412e9b76407c7cba8f92 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:26.643110 10586 raft_consensus.cc:3060] T c3112c3c3629412e9b76407c7cba8f92 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.642499 9932 consensus_queue.cc:260] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:26.643816 9931 tablet_bootstrap.cc:492] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:26.644237 9931 ts_tablet_manager.cc:1403] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.089s user 0.008s sys 0.001s
I20260504 13:56:26.644747 9932 ts_tablet_manager.cc:1434] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.023s user 0.001s sys 0.003s
I20260504 13:56:26.647123 9931 raft_consensus.cc:359] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.656663 9931 raft_consensus.cc:385] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.656973 9931 raft_consensus.cc:740] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.658704 9931 consensus_queue.cc:260] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.663756 10611 consensus_queue.cc:1048] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:26.668788 9931 ts_tablet_manager.cc:1434] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20260504 13:56:26.669862 9932 tablet_bootstrap.cc:492] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:26.670714 9931 tablet_bootstrap.cc:492] T 975fa1a4975242a388c15fe73abea1c7 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:26.680517 9932 tablet_bootstrap.cc:654] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.689400 9932 tablet_bootstrap.cc:492] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:26.689785 9932 ts_tablet_manager.cc:1403] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.000s
I20260504 13:56:26.692791 9931 tablet_bootstrap.cc:654] T 975fa1a4975242a388c15fe73abea1c7 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.694641 9932 raft_consensus.cc:359] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:26.695247 9932 raft_consensus.cc:385] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.695154 10611 raft_consensus.cc:493] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5beceb6317444247b31b44a6581707c6)
I20260504 13:56:26.695497 9932 raft_consensus.cc:740] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.695647 10611 raft_consensus.cc:515] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:26.697690 10611 leader_election.cc:290] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:26.697665 9932 consensus_queue.cc:260] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:26.699666 9473 leader_election.cc:343] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 48b193e958b4414aad29dd2bcee1a61b
W20260504 13:56:26.701891 9474 leader_election.cc:343] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Not found: Tablet not found: 48b193e958b4414aad29dd2bcee1a61b
I20260504 13:56:26.702282 9474 leader_election.cc:304] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:26.702674 9932 ts_tablet_manager.cc:1434] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260504 13:56:26.703502 9932 tablet_bootstrap.cc:492] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:26.703792 10566 raft_consensus.cc:2749] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:26.712733 9932 tablet_bootstrap.cc:654] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.713042 10586 raft_consensus.cc:2749] T c3112c3c3629412e9b76407c7cba8f92 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:26.715571 10629 raft_consensus.cc:515] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.718672 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5254b1061e2b424487f5d309baaee6b6" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:26.723490 9830 raft_consensus.cc:2393] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 1: Already voted for candidate d4051d2bf0bf410eb8acd107560df888 in this term.
I20260504 13:56:26.724014 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 152) took 4630 ms. Trace:
I20260504 13:56:26.730257 10391 rpcz_store.cc:276] 0504 13:56:22.093628 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:22.134608 (+ 40980us) service_pool.cc:224] Handling call
0504 13:56:26.285826 (+4151218us) ts_tablet_manager.cc:1151] Deleting tablet 3c43ff1d759a40d68344f25f066416ac
0504 13:56:26.285865 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:26.495607 (+209742us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:26.549289 (+ 53682us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:26.723942 (+174653us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":114625,"tablet-delete.queue_time_us":4075827}
I20260504 13:56:26.731544 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 3c43ff1d759a40d68344f25f066416ac (table default.Table_1f5a66e30f35482dab7c556daebc87c9 [id=c268fa33d24540b4978e3feebb0f793f]) successfully deleted
I20260504 13:56:26.733383 10013 tablet_bootstrap.cc:492] T c8989cdde4ca4e349079596e33723b2c P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:26.733811 10013 ts_tablet_manager.cc:1403] T c8989cdde4ca4e349079596e33723b2c P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.219s user 0.032s sys 0.052s
I20260504 13:56:26.735529 10611 raft_consensus.cc:493] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:26.735924 10611 raft_consensus.cc:515] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.740957 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "930078a10afa4dd987ad021cf9609514" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:26.741525 9830 raft_consensus.cc:2468] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:26.742544 9474 leader_election.cc:304] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:26.736537 10013 raft_consensus.cc:359] T c8989cdde4ca4e349079596e33723b2c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.746403 9932 tablet_bootstrap.cc:492] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:26.746373 10013 raft_consensus.cc:385] T c8989cdde4ca4e349079596e33723b2c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.746796 9932 ts_tablet_manager.cc:1403] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.001s
I20260504 13:56:26.746701 10013 raft_consensus.cc:740] T c8989cdde4ca4e349079596e33723b2c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.748236 10611 leader_election.cc:290] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.748093 10013 consensus_queue.cc:260] T c8989cdde4ca4e349079596e33723b2c P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.748652 10611 raft_consensus.cc:2804] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:26.748941 10611 raft_consensus.cc:493] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:26.749234 10611 raft_consensus.cc:3060] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.748987 9932 raft_consensus.cc:359] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.749490 9932 raft_consensus.cc:385] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.749712 9932 raft_consensus.cc:740] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.749877 10013 ts_tablet_manager.cc:1434] T c8989cdde4ca4e349079596e33723b2c P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.016s user 0.001s sys 0.005s
W20260504 13:56:26.752141 9473 leader_election.cc:343] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 930078a10afa4dd987ad021cf9609514
I20260504 13:56:26.754539 9931 tablet_bootstrap.cc:492] T 975fa1a4975242a388c15fe73abea1c7 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:26.754998 9931 ts_tablet_manager.cc:1403] T 975fa1a4975242a388c15fe73abea1c7 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.084s user 0.010s sys 0.003s
I20260504 13:56:26.754884 9932 consensus_queue.cc:260] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.759552 9932 ts_tablet_manager.cc:1434] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260504 13:56:26.760142 10431 raft_consensus.cc:515] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.761212 9537 raft_consensus.cc:2393] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
W20260504 13:56:26.766558 9328 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c47b6e3b4d0c45c28384f7aae6b749de consensus state has no leader
I20260504 13:56:26.775892 9932 tablet_bootstrap.cc:492] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:26.759414 9931 raft_consensus.cc:359] T 975fa1a4975242a388c15fe73abea1c7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.774073 10629 leader_election.cc:290] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.781509 9932 tablet_bootstrap.cc:654] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.779561 9931 raft_consensus.cc:385] T 975fa1a4975242a388c15fe73abea1c7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.781847 9931 raft_consensus.cc:740] T 975fa1a4975242a388c15fe73abea1c7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.782474 9931 consensus_queue.cc:260] T 975fa1a4975242a388c15fe73abea1c7 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.775079 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5254b1061e2b424487f5d309baaee6b6" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:26.783252 10013 tablet_bootstrap.cc:492] T 975fa1a4975242a388c15fe73abea1c7 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:26.783530 9681 raft_consensus.cc:3060] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.785068 9931 ts_tablet_manager.cc:1434] T 975fa1a4975242a388c15fe73abea1c7 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260504 13:56:26.787113 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5254b1061e2b424487f5d309baaee6b6" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:26.789489 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5254b1061e2b424487f5d309baaee6b6" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:26.789932 10431 leader_election.cc:290] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.790135 9830 raft_consensus.cc:2393] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 1: Already voted for candidate d4051d2bf0bf410eb8acd107560df888 in this term.
W20260504 13:56:26.791005 9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20260504 13:56:26.792052 9931 tablet_bootstrap.cc:492] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:26.791558 9474 leader_election.cc:304] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:26.792922 10391 tablet_replica.cc:333] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:26.800483 10391 raft_consensus.cc:2243] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:26.801003 10431 raft_consensus.cc:2749] T 5254b1061e2b424487f5d309baaee6b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260504 13:56:26.801819 9932 tablet_bootstrap.cc:492] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:26.802198 9932 ts_tablet_manager.cc:1403] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.005s
I20260504 13:56:26.801163 10668 raft_consensus.cc:493] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:26.802582 10668 raft_consensus.cc:515] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.804755 10668 leader_election.cc:290] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.805145 10668 raft_consensus.cc:493] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5beceb6317444247b31b44a6581707c6)
I20260504 13:56:26.805473 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb6927537704cc4b17617cbdaa21ac1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:26.805519 10668 raft_consensus.cc:515] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.806548 9684 raft_consensus.cc:2468] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:26.806835 9681 raft_consensus.cc:2468] T 5254b1061e2b424487f5d309baaee6b6 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:26.801684 10391 pending_rounds.cc:70] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00: Trying to abort 1 pending ops.
I20260504 13:56:26.807796 10391 pending_rounds.cc:77] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290606526533632 op_type: NO_OP noop_request { }
I20260504 13:56:26.807672 9473 leader_election.cc:304] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:26.807911 9768 leader_election.cc:304] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:26.808168 10391 raft_consensus.cc:2889] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:26.808483 10391 raft_consensus.cc:2272] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:26.810025 9473 leader_election.cc:343] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: bd8cd1caaeb249e3bafee7f1f0d4df2f
I20260504 13:56:26.810261 10391 ts_tablet_manager.cc:1916] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:26.810348 10629 raft_consensus.cc:2804] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:26.810681 10669 raft_consensus.cc:2804] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:26.810781 10629 raft_consensus.cc:697] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:26.811136 10669 raft_consensus.cc:493] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:26.811863 10669 raft_consensus.cc:3060] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.814155 10629 consensus_queue.cc:237] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.815496 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb6927537704cc4b17617cbdaa21ac1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:26.807693 10668 leader_election.cc:290] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.816047 9830 raft_consensus.cc:2468] T 7cb6927537704cc4b17617cbdaa21ac1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:26.817041 10586 raft_consensus.cc:493] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5beceb6317444247b31b44a6581707c6)
I20260504 13:56:26.817490 10586 raft_consensus.cc:515] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.819880 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd8cd1caaeb249e3bafee7f1f0d4df2f" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:26.820472 9830 raft_consensus.cc:2468] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:26.819990 10586 leader_election.cc:290] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:26.828526 9173 leader_election.cc:336] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337): Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111)
I20260504 13:56:26.828989 9173 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 806407748d564439807fe5d73db9abda; no voters: 6caedf407d13459e8fd477e8b60b7dc7, ec9f4c3a43d84fc9a420dd10723c01a1
W20260504 13:56:26.831009 9768 leader_election.cc:343] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: bd8cd1caaeb249e3bafee7f1f0d4df2f
I20260504 13:56:26.832628 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd8cd1caaeb249e3bafee7f1f0d4df2f" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:26.833134 9536 raft_consensus.cc:2468] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:26.835307 10668 raft_consensus.cc:493] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4051d2bf0bf410eb8acd107560df888)
I20260504 13:56:26.836061 9474 leader_election.cc:304] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:26.836719 10566 raft_consensus.cc:2804] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 13:56:26.837158 10566 raft_consensus.cc:493] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5beceb6317444247b31b44a6581707c6)
I20260504 13:56:26.835705 10668 raft_consensus.cc:515] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:26.837571 10566 raft_consensus.cc:3060] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Advancing to term 2
I20260504 13:56:26.838871 10013 tablet_bootstrap.cc:654] T 975fa1a4975242a388c15fe73abea1c7 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.840389 9931 tablet_bootstrap.cc:654] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.841794 9767 leader_election.cc:304] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:26.842840 10586 raft_consensus.cc:2804] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 13:56:26.843190 10586 raft_consensus.cc:493] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5beceb6317444247b31b44a6581707c6)
I20260504 13:56:26.843523 10586 raft_consensus.cc:3060] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Advancing to term 2
I20260504 13:56:26.846532 10571 raft_consensus.cc:493] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:26.846969 10571 raft_consensus.cc:515] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.849181 10571 leader_election.cc:290] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:26.849399 10611 raft_consensus.cc:515] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.850538 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "975fa1a4975242a388c15fe73abea1c7" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:26.850860 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "975fa1a4975242a388c15fe73abea1c7" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:26.851454 9830 raft_consensus.cc:2468] T 975fa1a4975242a388c15fe73abea1c7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
W20260504 13:56:26.851634 9618 leader_election.cc:343] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:26.852308 10600 raft_consensus.cc:493] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:26.852903 9620 leader_election.cc:304] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:26.852735 10600 raft_consensus.cc:515] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.854724 10600 leader_election.cc:290] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.855337 10571 raft_consensus.cc:2804] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:26.855652 10571 raft_consensus.cc:493] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:26.856037 10571 raft_consensus.cc:3060] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:26.856446 9473 leader_election.cc:343] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: c34877a81d504f46a2f680c5d5ddb4f4
I20260504 13:56:26.857411 10668 leader_election.cc:290] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:26.860870 10669 raft_consensus.cc:515] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.863246 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb6927537704cc4b17617cbdaa21ac1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:26.863125 10566 raft_consensus.cc:515] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.864164 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "930078a10afa4dd987ad021cf9609514" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:26.864667 9830 raft_consensus.cc:3060] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.804407 9932 raft_consensus.cc:359] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.866052 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cb6927537704cc4b17617cbdaa21ac1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:26.866526 9833 raft_consensus.cc:3060] T 7cb6927537704cc4b17617cbdaa21ac1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:26.868261 9473 leader_election.cc:343] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 930078a10afa4dd987ad021cf9609514
I20260504 13:56:26.872174 9833 raft_consensus.cc:2468] T 7cb6927537704cc4b17617cbdaa21ac1 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:26.872494 9830 raft_consensus.cc:2468] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:26.863870 9681 raft_consensus.cc:3060] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.865564 9932 raft_consensus.cc:385] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.878552 9932 raft_consensus.cc:740] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
W20260504 13:56:26.878858 9474 leader_election.cc:343] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Not found: Tablet not found: c34877a81d504f46a2f680c5d5ddb4f4
I20260504 13:56:26.879567 9474 leader_election.cc:304] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:26.879658 9932 consensus_queue.cc:260] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.879886 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8989cdde4ca4e349079596e33723b2c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:26.880396 9536 raft_consensus.cc:2468] T c8989cdde4ca4e349079596e33723b2c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:26.880512 9474 leader_election.cc:304] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:26.881227 10431 raft_consensus.cc:2749] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 13:56:26.881990 10431 raft_consensus.cc:2804] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:26.882450 10431 raft_consensus.cc:697] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
W20260504 13:56:26.883016 9473 leader_election.cc:343] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: bd8cd1caaeb249e3bafee7f1f0d4df2f
I20260504 13:56:26.883574 10566 leader_election.cc:290] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.883220 10431 consensus_queue.cc:237] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.865962 10611 leader_election.cc:290] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.887115 10669 leader_election.cc:290] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.889170 9681 raft_consensus.cc:2468] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:26.890928 10571 raft_consensus.cc:515] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.895397 9931 tablet_bootstrap.cc:492] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:26.896121 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8989cdde4ca4e349079596e33723b2c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:26.896636 9684 raft_consensus.cc:2468] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:26.897153 9931 ts_tablet_manager.cc:1403] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.111s user 0.010s sys 0.002s
W20260504 13:56:26.898123 9328 consensus_peers.cc:597] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 -> Peer 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337): Couldn't send request to peer 6caedf407d13459e8fd477e8b60b7dc7. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.253:36337: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260504 13:56:26.898202 10586 raft_consensus.cc:515] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.899864 9768 leader_election.cc:304] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:26.902980 9931 raft_consensus.cc:359] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.905742 9931 raft_consensus.cc:385] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.906440 9931 raft_consensus.cc:740] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.907519 9931 consensus_queue.cc:260] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.909471 10600 raft_consensus.cc:2804] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:26.909773 10600 raft_consensus.cc:493] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:26.910063 10600 raft_consensus.cc:3060] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.917217 10600 raft_consensus.cc:515] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.921801 10013 tablet_bootstrap.cc:492] T 975fa1a4975242a388c15fe73abea1c7 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:26.922271 10013 ts_tablet_manager.cc:1403] T 975fa1a4975242a388c15fe73abea1c7 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.139s user 0.016s sys 0.000s
I20260504 13:56:26.922926 9474 leader_election.cc:304] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:26.923676 10668 raft_consensus.cc:2804] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:26.924146 10668 raft_consensus.cc:697] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:26.925124 10668 consensus_queue.cc:237] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.927908 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd8cd1caaeb249e3bafee7f1f0d4df2f" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:26.928748 9830 raft_consensus.cc:2393] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 2: Already voted for candidate d4051d2bf0bf410eb8acd107560df888 in this term.
I20260504 13:56:26.930648 10600 leader_election.cc:290] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.931792 9474 leader_election.cc:304] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:26.932078 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8989cdde4ca4e349079596e33723b2c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:26.932574 9536 raft_consensus.cc:3060] T c8989cdde4ca4e349079596e33723b2c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.936355 10571 leader_election.cc:290] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:26.937625 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "975fa1a4975242a388c15fe73abea1c7" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:26.939366 10654 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:26.939672 9536 raft_consensus.cc:2468] T c8989cdde4ca4e349079596e33723b2c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:26.942653 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8989cdde4ca4e349079596e33723b2c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:26.943216 9681 raft_consensus.cc:3060] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.943910 10566 raft_consensus.cc:2749] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 13:56:26.945556 10013 raft_consensus.cc:359] T 975fa1a4975242a388c15fe73abea1c7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.946233 10013 raft_consensus.cc:385] T 975fa1a4975242a388c15fe73abea1c7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:26.946544 10013 raft_consensus.cc:740] T 975fa1a4975242a388c15fe73abea1c7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:26.951505 9931 ts_tablet_manager.cc:1434] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
I20260504 13:56:26.952303 10391 ts_tablet_manager.cc:1929] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:26.952565 9931 tablet_bootstrap.cc:492] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:26.952636 10391 log.cc:1199] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/07a428049fdb4a68aae729400e2d9674
I20260504 13:56:26.956938 10586 leader_election.cc:290] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:26.957711 10013 consensus_queue.cc:260] T 975fa1a4975242a388c15fe73abea1c7 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:26.964094 9537 raft_consensus.cc:3060] T 975fa1a4975242a388c15fe73abea1c7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:26.965682 10013 ts_tablet_manager.cc:1434] T 975fa1a4975242a388c15fe73abea1c7 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.043s user 0.001s sys 0.005s
I20260504 13:56:26.966529 10013 tablet_bootstrap.cc:492] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:26.969238 10391 ts_tablet_manager.cc:1950] T 07a428049fdb4a68aae729400e2d9674 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:26.969951 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 153) took 4830 ms. Trace:
I20260504 13:56:26.970331 10391 rpcz_store.cc:276] 0504 13:56:22.139357 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:22.139494 (+ 137us) service_pool.cc:224] Handling call
0504 13:56:26.767203 (+4627709us) ts_tablet_manager.cc:1151] Deleting tablet 07a428049fdb4a68aae729400e2d9674
0504 13:56:26.784781 (+ 17578us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:26.937978 (+153197us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:26.952191 (+ 14213us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:26.969902 (+ 17711us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4542184}
I20260504 13:56:26.971450 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 07a428049fdb4a68aae729400e2d9674 (table default.Table_0082bd435ced4fdcaeafe8848d05f9bf [id=386315c5331647659a31dc8a2436d7ce]) successfully deleted
I20260504 13:56:26.971993 9931 tablet_bootstrap.cc:654] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.972291 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 154) took 4832 ms. Trace:
I20260504 13:56:26.972546 10391 rpcz_store.cc:276] 0504 13:56:22.140169 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:22.140282 (+ 113us) service_pool.cc:224] Handling call
0504 13:56:26.972083 (+4831801us) ts_tablet_manager.cc:1151] Deleting tablet 2b50573310a04f3694eeab2fef0fb540
0504 13:56:26.972117 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:26.972249 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4746832}
W20260504 13:56:26.973518 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 2b50573310a04f3694eeab2fef0fb540 because the tablet was not found. No further retry: Not found: Tablet not found: 2b50573310a04f3694eeab2fef0fb540
I20260504 13:56:26.975450 9681 raft_consensus.cc:2468] T c8989cdde4ca4e349079596e33723b2c P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:26.977815 10391 tablet_replica.cc:333] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:26.978142 10013 tablet_bootstrap.cc:654] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:26.978544 10391 raft_consensus.cc:2243] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:26.979010 10391 raft_consensus.cc:2272] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:26.980722 10391 ts_tablet_manager.cc:1916] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260504 13:56:26.994104 9473 consensus_peers.cc:597] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1501455a0af348e38746d100987afdd0. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:26.986974 9537 raft_consensus.cc:2468] T 975fa1a4975242a388c15fe73abea1c7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:26.996274 9932 ts_tablet_manager.cc:1434] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.194s user 0.002s sys 0.021s
I20260504 13:56:26.999133 9505 tablet_service.cc:1511] Processing CreateTablet for tablet 5cb30835150949f39f6e7e7563381763 (DEFAULT_TABLE table=default.Table_583272d3574a4ca78d056fe7d6b922bc [id=a117c93d36da4d1aab22a554af5259d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.000599 9505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cb30835150949f39f6e7e7563381763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.002192 9339 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 0be795f04dd54cf19869c937b91d0ddf (table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]). Expected version 1 got 0
I20260504 13:56:27.007030 9509 tablet_service.cc:1511] Processing CreateTablet for tablet dffddd4b89984c54bc26dcb4a9488525 (DEFAULT_TABLE table=default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.008152 9509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dffddd4b89984c54bc26dcb4a9488525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:26.998227 9618 leader_election.cc:304] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:27.003839 9662 tablet_service.cc:1511] Processing CreateTablet for tablet 5cb30835150949f39f6e7e7563381763 (DEFAULT_TABLE table=default.Table_583272d3574a4ca78d056fe7d6b922bc [id=a117c93d36da4d1aab22a554af5259d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.015594 9662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cb30835150949f39f6e7e7563381763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.016000 10681 raft_consensus.cc:2804] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:27.016463 10681 raft_consensus.cc:697] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:27.017247 10681 consensus_queue.cc:237] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.022116 9659 tablet_service.cc:1511] Processing CreateTablet for tablet dffddd4b89984c54bc26dcb4a9488525 (DEFAULT_TABLE table=default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568]), partition=RANGE (key) PARTITION UNBOUNDED
W20260504 13:56:27.022926 9473 consensus_peers.cc:597] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: baf07a970a7c4a35b3c849b06d6c88d0. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:27.023799 9659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dffddd4b89984c54bc26dcb4a9488525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.022346 9339 catalog_manager.cc:5671] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:27.025061 9339 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:27.028120 9512 tablet_service.cc:1511] Processing CreateTablet for tablet 7a89bac060214d30924398007dfa1fa1 (DEFAULT_TABLE table=default.Table_2f0fc59ee89c402cb8a0fc23341636e6 [id=00b5922717934e15993bafde9f2513e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.029661 9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a89bac060214d30924398007dfa1fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.029587 9658 tablet_service.cc:1511] Processing CreateTablet for tablet 7a89bac060214d30924398007dfa1fa1 (DEFAULT_TABLE table=default.Table_2f0fc59ee89c402cb8a0fc23341636e6 [id=00b5922717934e15993bafde9f2513e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.030588 9658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a89bac060214d30924398007dfa1fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.035682 10571 raft_consensus.cc:493] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:27.036120 10571 raft_consensus.cc:515] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.036510 9508 tablet_service.cc:1511] Processing CreateTablet for tablet 538c8c2412104c3e8b19394a1cc38818 (DEFAULT_TABLE table=default.Table_7dbd11258f34498e82a86f7e3c72f470 [id=7bf52f9bd2a042e59fc4ff612d9846e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.037508 9508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 538c8c2412104c3e8b19394a1cc38818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.038436 10571 leader_election.cc:290] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:27.041143 9660 tablet_service.cc:1511] Processing CreateTablet for tablet 538c8c2412104c3e8b19394a1cc38818 (DEFAULT_TABLE table=default.Table_7dbd11258f34498e82a86f7e3c72f470 [id=7bf52f9bd2a042e59fc4ff612d9846e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.042063 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 538c8c2412104c3e8b19394a1cc38818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.042055 10013 tablet_bootstrap.cc:492] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:27.042429 10013 ts_tablet_manager.cc:1403] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.006s
I20260504 13:56:27.044632 9931 tablet_bootstrap.cc:492] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:27.044998 9931 ts_tablet_manager.cc:1403] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.093s user 0.011s sys 0.001s
I20260504 13:56:27.047086 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a985338790e4887af38c49ba013417f" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:27.047608 9931 raft_consensus.cc:359] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:27.048111 9618 leader_election.cc:343] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:27.048408 9931 raft_consensus.cc:385] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:27.048702 9931 raft_consensus.cc:740] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:27.049553 9931 consensus_queue.cc:260] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.053934 9518 tablet_service.cc:1511] Processing CreateTablet for tablet 89edd71cf5aa4a6abe2084695d3ed807 (DEFAULT_TABLE table=default.Table_69c9551ae2cf4ef399e40017a33bad73 [id=2d9b8dca85e9477eb77877d5618e9b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.054840 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89edd71cf5aa4a6abe2084695d3ed807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.060598 9931 ts_tablet_manager.cc:1434] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260504 13:56:27.064543 9511 tablet_service.cc:1511] Processing CreateTablet for tablet 3d9bd2722d23420ea0eac7664357a60c (DEFAULT_TABLE table=default.Table_821e9f466c974a2fbf24c07f8c266e62 [id=469bd5cb4e7f4a8a9c7af947a834ca7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.065101 10013 raft_consensus.cc:359] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.065629 10013 raft_consensus.cc:385] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:27.065665 9511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d9bd2722d23420ea0eac7664357a60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.065896 10013 raft_consensus.cc:740] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
W20260504 13:56:27.071282 10086 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 0.806s user 0.001s sys 0.011s
W20260504 13:56:27.071547 10086 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 0.807s user 0.001s sys 0.011s
I20260504 13:56:27.079098 9663 tablet_service.cc:1511] Processing CreateTablet for tablet 89edd71cf5aa4a6abe2084695d3ed807 (DEFAULT_TABLE table=default.Table_69c9551ae2cf4ef399e40017a33bad73 [id=2d9b8dca85e9477eb77877d5618e9b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.080374 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89edd71cf5aa4a6abe2084695d3ed807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.085577 9932 tablet_bootstrap.cc:492] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:27.066886 10013 consensus_queue.cc:260] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.096454 9341 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 7c9030802900499a90c4dbaf5c6936a4
I20260504 13:56:27.096819 10013 ts_tablet_manager.cc:1434] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.054s user 0.000s sys 0.006s
I20260504 13:56:27.097612 10013 tablet_bootstrap.cc:492] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:27.097689 9341 catalog_manager.cc:6741] ReplaceTablet: tablet 7c9030802900499a90c4dbaf5c6936a4 deleted and replaced by tablet 6dd139c347f7406aac340cc6ac994273
I20260504 13:56:27.098011 9341 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:35768 (request call id 256) took 1695 ms. Trace:
I20260504 13:56:27.098284 9341 rpcz_store.cc:276] 0504 13:56:25.403033 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:25.515814 (+112781us) service_pool.cc:224] Handling call
0504 13:56:27.097959 (+1582145us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:26.988373 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:26.988791 (+ 418us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:26.988817 (+ 26us) write_op.cc:435] Acquired schema lock
0504 13:56:26.988831 (+ 14us) tablet.cc:662] Decoding operations
0504 13:56:26.989191 (+ 360us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:26.989243 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:26.989269 (+ 26us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:26.989491 (+ 222us) tablet.cc:701] Row locks acquired
0504 13:56:26.989509 (+ 18us) write_op.cc:260] PREPARE: finished
0504 13:56:26.989632 (+ 123us) write_op.cc:270] Start()
0504 13:56:26.989721 (+ 89us) write_op.cc:276] Timestamp: P: 1777902986989594 usec, L: 0
0504 13:56:26.989737 (+ 16us) op_driver.cc:348] REPLICATION: starting
0504 13:56:26.990276 (+ 539us) log.cc:844] Serialized 628 byte log entry
0504 13:56:27.093213 (+102937us) op_driver.cc:464] REPLICATION: finished
0504 13:56:27.094326 (+ 1113us) write_op.cc:301] APPLY: starting
0504 13:56:27.094380 (+ 54us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:27.094481 (+ 101us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:27.094550 (+ 69us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:27.094859 (+ 309us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:27.095030 (+ 171us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:27.095093 (+ 63us) write_op.cc:312] APPLY: finished
0504 13:56:27.095326 (+ 233us) log.cc:844] Serialized 29 byte log entry
0504 13:56:27.095462 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:27.095524 (+ 62us) write_op.cc:454] Released schema lock
0504 13:56:27.095558 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:27.095763 (+ 205us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:27.095786 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":89,"apply.run_cpu_time_us":1413,"apply.run_wall_time_us":1414,"num_ops":2,"prepare.queue_time_us":88,"prepare.run_cpu_time_us":2230,"prepare.run_wall_time_us":2231,"raft.queue_time_us":1873,"raft.run_cpu_time_us":696,"raft.run_wall_time_us":698,"replication_time_us":103370}]]}
I20260504 13:56:27.101356 9509 tablet_service.cc:1558] Processing DeleteTablet for tablet 7c9030802900499a90c4dbaf5c6936a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6dd139c347f7406aac340cc6ac994273) from {username='slave'} at 127.0.0.1:52642
W20260504 13:56:27.102206 9341 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:38691 (request call id 41) took 1217 ms (client timeout 1000 ms). Trace:
W20260504 13:56:27.102486 9341 rpcz_store.cc:269] 0504 13:56:25.884592 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.101890 (+1217298us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:27.102150 (+ 260us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 13:56:27.103353 9341 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 36) took 1039 ms (client timeout 1000 ms). Trace:
W20260504 13:56:27.103621 9341 rpcz_store.cc:269] 0504 13:56:26.064139 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.103231 (+1039092us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:27.103312 (+ 81us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260504 13:56:27.105036 9658 tablet_service.cc:1558] Processing DeleteTablet for tablet 7c9030802900499a90c4dbaf5c6936a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6dd139c347f7406aac340cc6ac994273) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:27.110445 9932 tablet_bootstrap.cc:654] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:27.112339 9338 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 21d6358b451d4b8d87a22e30e0780f44
I20260504 13:56:27.113672 9338 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35768 (request call id 253) took 2687 ms. Trace:
I20260504 13:56:27.117552 9665 tablet_service.cc:1511] Processing CreateTablet for tablet 3d9bd2722d23420ea0eac7664357a60c (DEFAULT_TABLE table=default.Table_821e9f466c974a2fbf24c07f8c266e62 [id=469bd5cb4e7f4a8a9c7af947a834ca7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.118520 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d9bd2722d23420ea0eac7664357a60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.120255 9657 tablet_service.cc:1511] Processing CreateTablet for tablet 92480654807d495b879576b0195871bd (DEFAULT_TABLE table=default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.121213 9657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92480654807d495b879576b0195871bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.125242 9518 tablet_service.cc:1511] Processing CreateTablet for tablet b6103d953ace4cb8be3c580b3316f87c (DEFAULT_TABLE table=default.Table_a992fb6e038e4408a781d6d8ead37f54 [id=937a02c37dbd45d5bc92979ceaee1c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.061522 9931 tablet_bootstrap.cc:492] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:27.127593 9662 tablet_service.cc:1558] Processing DeleteTablet for tablet 21d6358b451d4b8d87a22e30e0780f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:25 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:27.128451 10391 ts_tablet_manager.cc:1929] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20260504 13:56:27.128803 10391 log.cc:1199] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/c34877a81d504f46a2f680c5d5ddb4f4
I20260504 13:56:27.113972 9338 rpcz_store.cc:276] 0504 13:56:24.426699 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:25.405219 (+978520us) service_pool.cc:224] Handling call
0504 13:56:25.406310 (+ 1091us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:25.406878 (+ 568us) catalog_manager.cc:2815] Modifying in-memory table state
0504 13:56:25.406993 (+ 115us) catalog_manager.cc:2824] Locking tablets
0504 13:56:26.987717 (+1580724us) catalog_manager.cc:2838] Removing table and tablets from system table
0504 13:56:27.112001 (+124284us) catalog_manager.cc:2856] Removing table from by-name map
0504 13:56:27.112179 (+ 178us) catalog_manager.cc:2872] Committing in-memory state
0504 13:56:27.112260 (+ 81us) catalog_manager.cc:2876] Aborting table tasks
0504 13:56:27.113605 (+ 1345us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:26.990716 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:26.996008 (+ 5292us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:26.996043 (+ 35us) write_op.cc:435] Acquired schema lock
0504 13:56:26.996055 (+ 12us) tablet.cc:662] Decoding operations
0504 13:56:26.996411 (+ 356us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:26.996444 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:26.996471 (+ 27us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:26.996746 (+ 275us) tablet.cc:701] Row locks acquired
0504 13:56:26.996759 (+ 13us) write_op.cc:260] PREPARE: finished
0504 13:56:26.996870 (+ 111us) write_op.cc:270] Start()
0504 13:56:26.996952 (+ 82us) write_op.cc:276] Timestamp: P: 1777902986996834 usec, L: 0
0504 13:56:26.996966 (+ 14us) op_driver.cc:348] REPLICATION: starting
0504 13:56:27.004086 (+ 7120us) spinlock_profiling.cc:243] Waited 5.48 ms on lock 0x7b5400030b18. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27c60e964 00007fc27b853d19 00007fc27b80fa01 00007fc27b80f6da 00007fc27b893d45 00007fc27b8941e9 00007fc27bd069f1 00007fc27bd061c8 00007fc27bd0ae61 00007fc27bd0ae19 00007fc27bd0ada9
0504 13:56:27.004393 (+ 307us) log.cc:844] Serialized 731 byte log entry
0504 13:56:27.008335 (+ 3942us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b58000500d0. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b7f281c 00007fc27b7f9133 00007fc27b7f90b9 00007fc27b7f9049 00007fc27b7f9011 00007fc27b7f81dd 00000000005b3bc4 00000000005b39f9 00007fc2747dbab5 00007fc2747dde01
0504 13:56:27.110148 (+101813us) op_driver.cc:464] REPLICATION: finished
0504 13:56:27.110381 (+ 233us) write_op.cc:301] APPLY: starting
0504 13:56:27.110436 (+ 55us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:27.110536 (+ 100us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:27.110556 (+ 20us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:27.111260 (+ 704us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:27.111393 (+ 133us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:27.111439 (+ 46us) write_op.cc:312] APPLY: finished
0504 13:56:27.111603 (+ 164us) log.cc:844] Serialized 29 byte log entry
0504 13:56:27.111737 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:27.111798 (+ 61us) write_op.cc:454] Released schema lock
0504 13:56:27.111828 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:27.112043 (+ 215us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:27.112075 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":78,"apply.run_cpu_time_us":1638,"apply.run_wall_time_us":1639,"num_ops":2,"prepare.queue_time_us":1434,"prepare.run_cpu_time_us":3748,"prepare.run_wall_time_us":14147,"raft.queue_time_us":97,"raft.run_cpu_time_us":1506,"raft.run_wall_time_us":3700,"replication_time_us":113096,"spinlock_wait_cycles":17874560}]]}
I20260504 13:56:27.129474 10391 ts_tablet_manager.cc:1950] T c34877a81d504f46a2f680c5d5ddb4f4 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:27.130106 9518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6103d953ace4cb8be3c580b3316f87c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.139784 10431 raft_consensus.cc:493] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:27.140300 10431 raft_consensus.cc:515] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.142138 10431 leader_election.cc:290] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:27.143229 10013 tablet_bootstrap.cc:654] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:27.143946 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8db7c4732fa048f9950248f45d144c57" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:27.144584 9684 raft_consensus.cc:2393] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 1: Already voted for candidate d4051d2bf0bf410eb8acd107560df888 in this term.
I20260504 13:56:27.148419 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 155) took 5007 ms. Trace:
I20260504 13:56:27.148795 10391 rpcz_store.cc:276] 0504 13:56:22.140958 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:22.141062 (+ 104us) service_pool.cc:224] Handling call
0504 13:56:26.977680 (+4836618us) ts_tablet_manager.cc:1151] Deleting tablet c34877a81d504f46a2f680c5d5ddb4f4
0504 13:56:26.977719 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:27.072731 (+ 95012us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:27.128320 (+ 55589us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:27.148367 (+ 20047us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4752176}
I20260504 13:56:27.151947 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet c34877a81d504f46a2f680c5d5ddb4f4 (table default.Table_c6a898ba23b74b54bb4684e970b44742 [id=92ea89acf1fc427690ee0a61cbbb1647]) successfully deleted
I20260504 13:56:27.152884 10391 tablet_replica.cc:333] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:27.153568 10391 raft_consensus.cc:2243] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:27.159484 10391 raft_consensus.cc:2272] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:27.167781 9662 tablet_service.cc:1511] Processing CreateTablet for tablet 59f9115b286d44f0b7eac7c74275cb21 (DEFAULT_TABLE table=default.Table_5944297f8c104978bbc6913ecd8f47a4 [id=3f11131cece44f3ebf3cd0f1e69139ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.156888 9506 tablet_service.cc:1511] Processing CreateTablet for tablet 59f9115b286d44f0b7eac7c74275cb21 (DEFAULT_TABLE table=default.Table_5944297f8c104978bbc6913ecd8f47a4 [id=3f11131cece44f3ebf3cd0f1e69139ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.157539 9512 tablet_service.cc:1558] Processing DeleteTablet for tablet 21d6358b451d4b8d87a22e30e0780f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:25 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:27.169278 9510 tablet_service.cc:1511] Processing CreateTablet for tablet e53381ee0e7f4e9781b833b435bca831 (DEFAULT_TABLE table=default.Table_3e6fece0bfdb447dae80b4598b049417 [id=fb4e5ce5526e44de8effdd766f4c257f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.156710 9509 tablet_service.cc:1511] Processing CreateTablet for tablet 92480654807d495b879576b0195871bd (DEFAULT_TABLE table=default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.174882 10391 ts_tablet_manager.cc:1916] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:27.175661 9662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59f9115b286d44f0b7eac7c74275cb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.177245 10013 tablet_bootstrap.cc:492] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:27.178483 10013 ts_tablet_manager.cc:1403] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.081s user 0.005s sys 0.005s
I20260504 13:56:27.179965 9666 tablet_service.cc:1511] Processing CreateTablet for tablet e53381ee0e7f4e9781b833b435bca831 (DEFAULT_TABLE table=default.Table_3e6fece0bfdb447dae80b4598b049417 [id=fb4e5ce5526e44de8effdd766f4c257f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.180855 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e53381ee0e7f4e9781b833b435bca831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.180790 10013 raft_consensus.cc:359] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.181381 10013 raft_consensus.cc:385] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:27.181612 10013 raft_consensus.cc:740] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:27.183871 9508 tablet_service.cc:1511] Processing CreateTablet for tablet 982f7c53a1f0482e862b53aeeee7cd25 (DEFAULT_TABLE table=default.Table_1e6b4c9e684e45b3b2510d3979c1b5c4 [id=ecfa480ab0ea40008ff4507ed6c5b964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.188993 9335 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet fe7ab3b577b74a9c9a40cb15c94f8aa1 (table default.Table_dee1335eff89438e9c68835a4c258031 [id=1714df2b31ff4865b6b357500d121a26]). Expected version 1 got 0
I20260504 13:56:27.190259 9335 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 0be795f04dd54cf19869c937b91d0ddf (table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]). Expected version 1 got 0
I20260504 13:56:27.197046 10691 tablet_replica.cc:333] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:27.197562 9658 tablet_service.cc:1511] Processing CreateTablet for tablet 982f7c53a1f0482e862b53aeeee7cd25 (DEFAULT_TABLE table=default.Table_1e6b4c9e684e45b3b2510d3979c1b5c4 [id=ecfa480ab0ea40008ff4507ed6c5b964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.198062 10691 raft_consensus.cc:2243] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:27.198542 10691 raft_consensus.cc:2272] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:27.200661 10691 ts_tablet_manager.cc:1916] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:27.203747 9512 tablet_service.cc:1511] Processing CreateTablet for tablet b84c3bedc7a84e0d8a873fb15689b314 (DEFAULT_TABLE table=default.Table_7f912e1d4381428e9572cbb4b13f4eec [id=fb18c55d21704b2ab04dbf35d21377bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.207726 9520 tablet_service.cc:1511] Processing CreateTablet for tablet a862a43c5ac443d9bac98e0472ca23b2 (DEFAULT_TABLE table=default.Table_0eae04160cf241df95cc53da6f4dfacf [id=3822050e2a9d4a9db8b0e36430fd0b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.212894 10013 consensus_queue.cc:260] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.214684 10013 ts_tablet_manager.cc:1434] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.036s user 0.004s sys 0.000s
I20260504 13:56:27.219286 9335 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:27.221506 10431 raft_consensus.cc:493] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:27.221935 10431 raft_consensus.cc:515] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.225188 10431 leader_election.cc:290] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:27.227833 9506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59f9115b286d44f0b7eac7c74275cb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.231666 9510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e53381ee0e7f4e9781b833b435bca831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.237515 9507 tablet_service.cc:1511] Processing CreateTablet for tablet 7d27b04d3f4443c5ade9616d8626d50e (DEFAULT_TABLE table=default.Table_8ccb9fc27b374611b20008514d3ad483 [id=6c295b6b72c94ca0b5f1805a555e7f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.238319 9509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92480654807d495b879576b0195871bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.238898 9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b84c3bedc7a84e0d8a873fb15689b314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.243403 9520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a862a43c5ac443d9bac98e0472ca23b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.244496 9516 tablet_service.cc:1511] Processing CreateTablet for tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3 (DEFAULT_TABLE table=default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.245537 9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.247529 9507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d27b04d3f4443c5ade9616d8626d50e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.275584 9508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 982f7c53a1f0482e862b53aeeee7cd25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.277696 9517 tablet_service.cc:1511] Processing CreateTablet for tablet c1a01231bb044433ae5827d07a6592f8 (DEFAULT_TABLE table=default.Table_3387191fcd6046abbf257447bfbb27f7 [id=d195f5bef6ce49f1b87a4bf5f516eebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.278648 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1a01231bb044433ae5827d07a6592f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.280433 10013 tablet_bootstrap.cc:492] T 1fbe09b4078c4eebb52d7d00bc87371c P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:27.286211 10013 tablet_bootstrap.cc:654] T 1fbe09b4078c4eebb52d7d00bc87371c P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:27.288836 9505 tablet_service.cc:1511] Processing CreateTablet for tablet 88619eaa917d4c33a910fc1347ab126e (DEFAULT_TABLE table=default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.289852 9505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88619eaa917d4c33a910fc1347ab126e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.292415 9335 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 8db7c4732fa048f9950248f45d144c57 (table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62]). Expected version 1 got 0
I20260504 13:56:27.198553 9658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 982f7c53a1f0482e862b53aeeee7cd25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.206485 9656 tablet_service.cc:1511] Processing CreateTablet for tablet b84c3bedc7a84e0d8a873fb15689b314 (DEFAULT_TABLE table=default.Table_7f912e1d4381428e9572cbb4b13f4eec [id=fb18c55d21704b2ab04dbf35d21377bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.298776 9656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b84c3bedc7a84e0d8a873fb15689b314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.293006 9335 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 1fbe09b4078c4eebb52d7d00bc87371c (table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee]). Expected version 1 got 0
I20260504 13:56:27.215457 9660 tablet_service.cc:1511] Processing CreateTablet for tablet b6103d953ace4cb8be3c580b3316f87c (DEFAULT_TABLE table=default.Table_a992fb6e038e4408a781d6d8ead37f54 [id=937a02c37dbd45d5bc92979ceaee1c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.312196 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6103d953ace4cb8be3c580b3316f87c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.312875 9335 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]). Expected version 2 got 0
I20260504 13:56:27.217422 9661 tablet_service.cc:1511] Processing CreateTablet for tablet a862a43c5ac443d9bac98e0472ca23b2 (DEFAULT_TABLE table=default.Table_0eae04160cf241df95cc53da6f4dfacf [id=3822050e2a9d4a9db8b0e36430fd0b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.314815 9661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a862a43c5ac443d9bac98e0472ca23b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.224813 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8702ce3ff1a845f3b70012855cd17efd" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:27.319535 9684 raft_consensus.cc:2468] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:27.319832 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd8cd1caaeb249e3bafee7f1f0d4df2f" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:27.320528 9537 raft_consensus.cc:2393] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 2: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:27.323212 10611 raft_consensus.cc:493] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5beceb6317444247b31b44a6581707c6)
I20260504 13:56:27.323676 10611 raft_consensus.cc:515] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:27.237233 9932 tablet_bootstrap.cc:492] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:27.287901 9665 tablet_service.cc:1511] Processing CreateTablet for tablet c1a01231bb044433ae5827d07a6592f8 (DEFAULT_TABLE table=default.Table_3387191fcd6046abbf257447bfbb27f7 [id=d195f5bef6ce49f1b87a4bf5f516eebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.325421 9932 ts_tablet_manager.cc:1403] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.252s user 0.006s sys 0.006s
I20260504 13:56:27.325884 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1a01231bb044433ae5827d07a6592f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.325830 10611 leader_election.cc:290] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:27.327826 9932 raft_consensus.cc:359] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.328404 9932 raft_consensus.cc:385] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:27.328641 9932 raft_consensus.cc:740] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:27.329186 9932 consensus_queue.cc:260] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.330416 9473 leader_election.cc:304] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:27.331099 10611 raft_consensus.cc:2804] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:27.331462 10611 raft_consensus.cc:493] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:27.331800 10611 raft_consensus.cc:3060] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:27.335901 9659 tablet_service.cc:1511] Processing CreateTablet for tablet 7d27b04d3f4443c5ade9616d8626d50e (DEFAULT_TABLE table=default.Table_8ccb9fc27b374611b20008514d3ad483 [id=6c295b6b72c94ca0b5f1805a555e7f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.337038 9659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d27b04d3f4443c5ade9616d8626d50e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.349573 9666 tablet_service.cc:1511] Processing CreateTablet for tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3 (DEFAULT_TABLE table=default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.350694 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.324668 9767 leader_election.cc:304] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:27.335592 9813 tablet_service.cc:1511] Processing CreateTablet for tablet 59f9115b286d44f0b7eac7c74275cb21 (DEFAULT_TABLE table=default.Table_5944297f8c104978bbc6913ecd8f47a4 [id=3f11131cece44f3ebf3cd0f1e69139ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.366021 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59f9115b286d44f0b7eac7c74275cb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.367511 10431 consensus_queue.cc:1048] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:27.335592 9797 tablet_service.cc:1511] Processing CreateTablet for tablet c1a01231bb044433ae5827d07a6592f8 (DEFAULT_TABLE table=default.Table_3387191fcd6046abbf257447bfbb27f7 [id=d195f5bef6ce49f1b87a4bf5f516eebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.337325 9798 tablet_service.cc:1511] Processing CreateTablet for tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3 (DEFAULT_TABLE table=default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.338721 9799 tablet_service.cc:1511] Processing CreateTablet for tablet 7d27b04d3f4443c5ade9616d8626d50e (DEFAULT_TABLE table=default.Table_8ccb9fc27b374611b20008514d3ad483 [id=6c295b6b72c94ca0b5f1805a555e7f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.340149 9800 tablet_service.cc:1511] Processing CreateTablet for tablet a862a43c5ac443d9bac98e0472ca23b2 (DEFAULT_TABLE table=default.Table_0eae04160cf241df95cc53da6f4dfacf [id=3822050e2a9d4a9db8b0e36430fd0b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.341532 9801 tablet_service.cc:1511] Processing CreateTablet for tablet b84c3bedc7a84e0d8a873fb15689b314 (DEFAULT_TABLE table=default.Table_7f912e1d4381428e9572cbb4b13f4eec [id=fb18c55d21704b2ab04dbf35d21377bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.342746 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "975fa1a4975242a388c15fe73abea1c7" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:27.343189 9802 tablet_service.cc:1511] Processing CreateTablet for tablet 982f7c53a1f0482e862b53aeeee7cd25 (DEFAULT_TABLE table=default.Table_1e6b4c9e684e45b3b2510d3979c1b5c4 [id=ecfa480ab0ea40008ff4507ed6c5b964]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.369675 9802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 982f7c53a1f0482e862b53aeeee7cd25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.344583 10629 raft_consensus.cc:493] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:27.344916 9803 tablet_service.cc:1511] Processing CreateTablet for tablet b6103d953ace4cb8be3c580b3316f87c (DEFAULT_TABLE table=default.Table_a992fb6e038e4408a781d6d8ead37f54 [id=937a02c37dbd45d5bc92979ceaee1c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.345281 9810 tablet_service.cc:1511] Processing CreateTablet for tablet 89edd71cf5aa4a6abe2084695d3ed807 (DEFAULT_TABLE table=default.Table_69c9551ae2cf4ef399e40017a33bad73 [id=2d9b8dca85e9477eb77877d5618e9b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.346290 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8db7c4732fa048f9950248f45d144c57" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:27.372888 10629 raft_consensus.cc:515] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.346535 9804 tablet_service.cc:1511] Processing CreateTablet for tablet 538c8c2412104c3e8b19394a1cc38818 (DEFAULT_TABLE table=default.Table_7dbd11258f34498e82a86f7e3c72f470 [id=7bf52f9bd2a042e59fc4ff612d9846e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.347404 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a985338790e4887af38c49ba013417f" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:27.348289 9805 tablet_service.cc:1511] Processing CreateTablet for tablet 7a89bac060214d30924398007dfa1fa1 (DEFAULT_TABLE table=default.Table_2f0fc59ee89c402cb8a0fc23341636e6 [id=00b5922717934e15993bafde9f2513e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.350041 9806 tablet_service.cc:1511] Processing CreateTablet for tablet 5cb30835150949f39f6e7e7563381763 (DEFAULT_TABLE table=default.Table_583272d3574a4ca78d056fe7d6b922bc [id=a117c93d36da4d1aab22a554af5259d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.351557 9807 tablet_service.cc:1511] Processing CreateTablet for tablet dffddd4b89984c54bc26dcb4a9488525 (DEFAULT_TABLE table=default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.352672 9808 tablet_service.cc:1511] Processing CreateTablet for tablet 92480654807d495b879576b0195871bd (DEFAULT_TABLE table=default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.353781 9809 tablet_service.cc:1511] Processing CreateTablet for tablet 3d9bd2722d23420ea0eac7664357a60c (DEFAULT_TABLE table=default.Table_821e9f466c974a2fbf24c07f8c266e62 [id=469bd5cb4e7f4a8a9c7af947a834ca7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.354501 9811 tablet_service.cc:1558] Processing DeleteTablet for tablet 7c9030802900499a90c4dbaf5c6936a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6dd139c347f7406aac340cc6ac994273) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:27.355713 9812 tablet_service.cc:1511] Processing CreateTablet for tablet e53381ee0e7f4e9781b833b435bca831 (DEFAULT_TABLE table=default.Table_3e6fece0bfdb447dae80b4598b049417 [id=fb4e5ce5526e44de8effdd766f4c257f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.356477 9814 tablet_service.cc:1558] Processing DeleteTablet for tablet 21d6358b451d4b8d87a22e30e0780f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:25 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:27.364217 9823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030b35c3409143108c34ab1f0ee1b96d" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:27.383432 10391 ts_tablet_manager.cc:1929] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:27.379004 9830 raft_consensus.cc:3060] T 975fa1a4975242a388c15fe73abea1c7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:27.390147 9664 tablet_service.cc:1511] Processing CreateTablet for tablet 88619eaa917d4c33a910fc1347ab126e (DEFAULT_TABLE table=default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.395699 9664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88619eaa917d4c33a910fc1347ab126e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.396713 9932 ts_tablet_manager.cc:1434] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.071s user 0.003s sys 0.002s
I20260504 13:56:27.398591 9932 tablet_bootstrap.cc:492] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:27.407894 9620 leader_election.cc:343] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:27.408201 10681 raft_consensus.cc:493] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:27.408272 9620 leader_election.cc:304] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:27.408653 10681 raft_consensus.cc:515] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:27.393555 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 67646.1us
I20260504 13:56:27.410449 10681 leader_election.cc:290] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:27.410872 10681 raft_consensus.cc:2749] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:27.411300 10683 raft_consensus.cc:493] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:27.411701 10683 raft_consensus.cc:515] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.412940 10391 log.cc:1199] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/48b193e958b4414aad29dd2bcee1a61b
I20260504 13:56:27.413408 10683 leader_election.cc:290] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:27.413667 10391 ts_tablet_manager.cc:1950] T 48b193e958b4414aad29dd2bcee1a61b P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:27.414408 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 156) took 5016 ms. Trace:
I20260504 13:56:27.416695 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fbe09b4078c4eebb52d7d00bc87371c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:27.417819 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "184fefa651fc411fb5107cc923c02535" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:27.418995 9618 leader_election.cc:343] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:27.419540 9618 leader_election.cc:343] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:27.420943 9474 leader_election.cc:304] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:27.421828 10669 raft_consensus.cc:493] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:27.422214 10669 raft_consensus.cc:515] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.391104 10668 consensus_queue.cc:1048] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:27.423830 10668 raft_consensus.cc:3060] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:27.423947 10669 leader_election.cc:290] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:27.424419 9341 catalog_manager.cc:5671] T dd3c045379734134b49e94340679c11d P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:27.426643 9341 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:27.427891 9516 tablet_service.cc:1511] Processing CreateTablet for tablet b8ffeae4e5af4cc8a3eb3bc0bedf18cc (DEFAULT_TABLE table=default.Table_003cbc6fdf0b4dddaba008af167dc621 [id=4629003a2d8d461c9e309d49ddecea6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.429987 9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8ffeae4e5af4cc8a3eb3bc0bedf18cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.431254 9825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8702ce3ff1a845f3b70012855cd17efd" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:27.439867 10675 raft_consensus.cc:2804] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:27.441054 10586 raft_consensus.cc:493] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:27.441519 10586 raft_consensus.cc:515] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.444103 10611 raft_consensus.cc:515] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.446666 10611 leader_election.cc:290] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:27.450845 9657 tablet_service.cc:1511] Processing CreateTablet for tablet b8ffeae4e5af4cc8a3eb3bc0bedf18cc (DEFAULT_TABLE table=default.Table_003cbc6fdf0b4dddaba008af167dc621 [id=4629003a2d8d461c9e309d49ddecea6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.452308 9655 tablet_service.cc:1511] Processing CreateTablet for tablet af5c5a6cae4e4b6ea3eece2e21a587ec (DEFAULT_TABLE table=default.Table_cdee25533fe54da0a74b421714b0d7c4 [id=81df524d85774544807ba1902be1aae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.456023 10571 raft_consensus.cc:493] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:27.456436 10571 raft_consensus.cc:515] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:27.456756 9341 catalog_manager.cc:5671] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:27.458173 10571 leader_election.cc:290] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:27.459838 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274bb204041c4e1085395c7bd938aaf1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:27.462006 9618 leader_election.cc:343] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:27.395363 9473 leader_election.cc:343] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 030b35c3409143108c34ab1f0ee1b96d
W20260504 13:56:27.466742 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 71543.8us
I20260504 13:56:27.458105 9341 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 8db7c4732fa048f9950248f45d144c57 (table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62]). Expected version 1 got 0
I20260504 13:56:27.468079 9341 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 1fbe09b4078c4eebb52d7d00bc87371c (table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee]). Expected version 1 got 0
I20260504 13:56:27.468556 9341 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]). Expected version 2 got 0
I20260504 13:56:27.472685 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f3905bfb30e4b03a7fcf0a2f57e2117" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:27.473059 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8702ce3ff1a845f3b70012855cd17efd" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:27.473249 9681 raft_consensus.cc:2468] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:27.473516 9686 raft_consensus.cc:3060] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:27.474426 9473 leader_election.cc:304] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:27.475028 10669 raft_consensus.cc:2804] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:27.475373 10669 raft_consensus.cc:493] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:27.475706 10669 raft_consensus.cc:3060] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:27.476449 9507 tablet_service.cc:1511] Processing CreateTablet for tablet a2bfb681e6824e9f9bdcbea23061d57a (DEFAULT_TABLE table=default.Table_e55147137c794be4bff21afce5d36bbc [id=b45733fe3c5547bea00e3967be81ca35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.477444 9507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2bfb681e6824e9f9bdcbea23061d57a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.483479 10669 raft_consensus.cc:515] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.414666 10391 rpcz_store.cc:276] 0504 13:56:22.412475 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:22.413100 (+ 625us) service_pool.cc:224] Handling call
0504 13:56:27.152748 (+4739648us) ts_tablet_manager.cc:1151] Deleting tablet 48b193e958b4414aad29dd2bcee1a61b
0504 13:56:27.152787 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:27.210192 (+ 57405us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:27.383322 (+173130us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:27.414361 (+ 31039us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4739107}
I20260504 13:56:27.485867 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 48b193e958b4414aad29dd2bcee1a61b (table default.Table_2c39b14515bb46f6a0b0f1252d8e273d [id=e9e56499fd2243919308927b3614ae8e]) successfully deleted
I20260504 13:56:27.486706 10391 tablet_replica.cc:333] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:27.487442 10391 raft_consensus.cc:2243] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:27.487874 10391 raft_consensus.cc:2272] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:27.488987 10668 raft_consensus.cc:2749] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:27.489027 9686 raft_consensus.cc:2468] T 8702ce3ff1a845f3b70012855cd17efd P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:27.490598 9473 leader_election.cc:304] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:27.491350 10431 raft_consensus.cc:2804] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:27.491850 10431 raft_consensus.cc:697] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:27.492370 10013 tablet_bootstrap.cc:492] T 1fbe09b4078c4eebb52d7d00bc87371c P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:27.492741 10013 ts_tablet_manager.cc:1403] T 1fbe09b4078c4eebb52d7d00bc87371c P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.212s user 0.014s sys 0.015s
I20260504 13:56:27.492651 10431 consensus_queue.cc:237] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.459089 9932 tablet_bootstrap.cc:654] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:27.495074 10013 raft_consensus.cc:359] T 1fbe09b4078c4eebb52d7d00bc87371c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.495898 10013 raft_consensus.cc:385] T 1fbe09b4078c4eebb52d7d00bc87371c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:27.496201 10013 raft_consensus.cc:740] T 1fbe09b4078c4eebb52d7d00bc87371c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
W20260504 13:56:27.497203 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 85247.4us
I20260504 13:56:27.500727 10629 leader_election.cc:290] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:27.500675 9509 tablet_service.cc:1511] Processing CreateTablet for tablet af5c5a6cae4e4b6ea3eece2e21a587ec (DEFAULT_TABLE table=default.Table_cdee25533fe54da0a74b421714b0d7c4 [id=81df524d85774544807ba1902be1aae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.501683 9509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af5c5a6cae4e4b6ea3eece2e21a587ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.504472 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f3905bfb30e4b03a7fcf0a2f57e2117" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:27.505203 9536 raft_consensus.cc:2393] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:27.506385 10669 leader_election.cc:290] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:27.508939 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f3905bfb30e4b03a7fcf0a2f57e2117" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:27.510072 9686 raft_consensus.cc:3060] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:27.514454 9655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af5c5a6cae4e4b6ea3eece2e21a587ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.469472 10586 leader_election.cc:290] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:27.540201 10691 ts_tablet_manager.cc:1929] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:27.540536 10691 log.cc:1199] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/7c9030802900499a90c4dbaf5c6936a4
I20260504 13:56:27.541167 10691 ts_tablet_manager.cc:1950] T 7c9030802900499a90c4dbaf5c6936a4 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:27.541706 9657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8ffeae4e5af4cc8a3eb3bc0bedf18cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.543762 9665 tablet_service.cc:1511] Processing CreateTablet for tablet a2bfb681e6824e9f9bdcbea23061d57a (DEFAULT_TABLE table=default.Table_e55147137c794be4bff21afce5d36bbc [id=b45733fe3c5547bea00e3967be81ca35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.544744 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2bfb681e6824e9f9bdcbea23061d57a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.556339 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274bb204041c4e1085395c7bd938aaf1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:27.556766 9814 tablet_service.cc:1511] Processing CreateTablet for tablet 88619eaa917d4c33a910fc1347ab126e (DEFAULT_TABLE table=default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.559240 9807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dffddd4b89984c54bc26dcb4a9488525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.563403 9804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 538c8c2412104c3e8b19394a1cc38818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.567606 9809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d9bd2722d23420ea0eac7664357a60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.568221 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "184fefa651fc411fb5107cc923c02535" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:27.570323 9620 leader_election.cc:343] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:27.570691 9620 leader_election.cc:304] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:27.571878 10571 raft_consensus.cc:2749] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:27.572891 9620 leader_election.cc:343] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:27.573247 9620 leader_election.cc:304] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:27.573817 10571 raft_consensus.cc:2749] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:27.576879 10013 consensus_queue.cc:260] T 1fbe09b4078c4eebb52d7d00bc87371c P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.588829 10391 ts_tablet_manager.cc:1916] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:27.557457 10675 raft_consensus.cc:697] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:27.605124 9474 leader_election.cc:304] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:27.605939 9474 raft_consensus.cc:2686] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260504 13:56:27.607199 9474 leader_election.cc:343] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:27.613253 9823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f3905bfb30e4b03a7fcf0a2f57e2117" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:27.613790 9823 raft_consensus.cc:3060] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:27.615162 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8702ce3ff1a845f3b70012855cd17efd" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:27.619736 10571 raft_consensus.cc:493] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:27.620157 10571 raft_consensus.cc:515] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:27.621912 10571 leader_election.cc:290] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:27.622596 9932 tablet_bootstrap.cc:492] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:27.621809 9338 catalog_manager.cc:5671] T c3112c3c3629412e9b76407c7cba8f92 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:27.622969 9932 ts_tablet_manager.cc:1403] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.225s user 0.013s sys 0.000s
I20260504 13:56:27.623265 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60d3cc59ffe947fbb57b7d3301ccbac1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:27.624904 9618 leader_election.cc:343] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:27.624933 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 7c9030802900499a90c4dbaf5c6936a4 (table default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]) successfully deleted
I20260504 13:56:27.579051 10013 ts_tablet_manager.cc:1434] T 1fbe09b4078c4eebb52d7d00bc87371c P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.086s user 0.004s sys 0.003s
W20260504 13:56:27.625506 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 21d6358b451d4b8d87a22e30e0780f44 because the tablet was not found. No further retry: Not found: Tablet not found: 21d6358b451d4b8d87a22e30e0780f44
I20260504 13:56:27.625936 10013 tablet_bootstrap.cc:492] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:27.641556 9932 raft_consensus.cc:359] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.643326 9824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f3905bfb30e4b03a7fcf0a2f57e2117" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:27.642318 9932 raft_consensus.cc:385] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:27.645370 10013 tablet_bootstrap.cc:654] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:27.645550 9932 raft_consensus.cc:740] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:27.646234 9799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d27b04d3f4443c5ade9616d8626d50e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.648504 9932 consensus_queue.cc:260] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.650601 9505 tablet_service.cc:1511] Processing CreateTablet for tablet 99e819494171485e9af8bc8ec33a6ff0 (DEFAULT_TABLE table=default.Table_89c38f9ba46e48fab397a7f3d5d1b805 [id=e4f6621892874f979d1c579d487cba44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.652016 9505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99e819494171485e9af8bc8ec33a6ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.652596 9830 raft_consensus.cc:2468] T 975fa1a4975242a388c15fe73abea1c7 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:27.656076 9811 tablet_service.cc:1511] Processing CreateTablet for tablet b8ffeae4e5af4cc8a3eb3bc0bedf18cc (DEFAULT_TABLE table=default.Table_003cbc6fdf0b4dddaba008af167dc621 [id=4629003a2d8d461c9e309d49ddecea6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.656360 10013 tablet_bootstrap.cc:492] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:27.656702 10013 ts_tablet_manager.cc:1403] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.004s
I20260504 13:56:27.658856 10013 raft_consensus.cc:359] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.659389 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88619eaa917d4c33a910fc1347ab126e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.659471 10013 raft_consensus.cc:385] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:27.659704 10013 raft_consensus.cc:740] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:27.660377 10013 consensus_queue.cc:260] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.662010 9931 tablet_bootstrap.cc:654] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:27.662540 10013 ts_tablet_manager.cc:1434] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260504 13:56:27.664125 10013 tablet_bootstrap.cc:492] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:27.595705 10675 consensus_queue.cc:237] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.669274 10013 tablet_bootstrap.cc:654] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:27.676627 9823 raft_consensus.cc:2468] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:27.679018 9810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89edd71cf5aa4a6abe2084695d3ed807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.681166 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fbe09b4078c4eebb52d7d00bc87371c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:27.681725 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 21d6358b451d4b8d87a22e30e0780f44 because the tablet was not found. No further retry: Not found: Tablet not found: 21d6358b451d4b8d87a22e30e0780f44
I20260504 13:56:27.681761 9536 raft_consensus.cc:2468] T 1fbe09b4078c4eebb52d7d00bc87371c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:27.684590 9796 tablet_service.cc:1511] Processing CreateTablet for tablet af5c5a6cae4e4b6ea3eece2e21a587ec (DEFAULT_TABLE table=default.Table_cdee25533fe54da0a74b421714b0d7c4 [id=81df524d85774544807ba1902be1aae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.685046 9811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8ffeae4e5af4cc8a3eb3bc0bedf18cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.686249 10697 consensus_queue.cc:1048] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260504 13:56:27.696697 9795 tablet_service.cc:1511] Processing CreateTablet for tablet a2bfb681e6824e9f9bdcbea23061d57a (DEFAULT_TABLE table=default.Table_e55147137c794be4bff21afce5d36bbc [id=b45733fe3c5547bea00e3967be81ca35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.697399 9666 tablet_service.cc:1511] Processing CreateTablet for tablet 99e819494171485e9af8bc8ec33a6ff0 (DEFAULT_TABLE table=default.Table_89c38f9ba46e48fab397a7f3d5d1b805 [id=e4f6621892874f979d1c579d487cba44]), partition=RANGE (key) PARTITION UNBOUNDED
W20260504 13:56:27.705295 9474 leader_election.cc:343] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:27.705525 9932 ts_tablet_manager.cc:1434] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.074s user 0.005s sys 0.001s
I20260504 13:56:27.709437 9686 raft_consensus.cc:2468] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:27.731163 9473 leader_election.cc:304] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:27.732100 10703 raft_consensus.cc:2804] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:27.710783 9338 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 8db7c4732fa048f9950248f45d144c57 (table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62]). Expected version 1 got 0
I20260504 13:56:27.732854 10703 raft_consensus.cc:697] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:27.733736 10703 consensus_queue.cc:237] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.746682 9655 tablet_service.cc:1511] Processing CreateTablet for tablet 36c8e9ba29a1466082a3e8bedec87b91 (DEFAULT_TABLE table=default.Table_1665799e268a496d9ffc5568fa34a763 [id=403cbbc8a1304e5695a89d3697932c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.750730 9338 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:27.754420 9517 tablet_service.cc:1511] Processing CreateTablet for tablet cc2f95dadd544f57bd6d1467d30e954d (DEFAULT_TABLE table=default.Table_2585508fc4214c8b82d2957644309ac2 [id=256b605e5a2342ed883ec30e3de0ffda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.755659 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc2f95dadd544f57bd6d1467d30e954d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.759341 9338 catalog_manager.cc:5671] T 7ed3ee9a13e84594905db47aa33040c2 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:27.762748 9665 tablet_service.cc:1511] Processing CreateTablet for tablet cc2f95dadd544f57bd6d1467d30e954d (DEFAULT_TABLE table=default.Table_2585508fc4214c8b82d2957644309ac2 [id=256b605e5a2342ed883ec30e3de0ffda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.765394 9508 tablet_service.cc:1511] Processing CreateTablet for tablet 2ddf890c17964e7f90409ebe3c4034ef (DEFAULT_TABLE table=default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.766466 9508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ddf890c17964e7f90409ebe3c4034ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.768847 9338 catalog_manager.cc:5671] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:27.772522 9656 tablet_service.cc:1511] Processing CreateTablet for tablet 2ddf890c17964e7f90409ebe3c4034ef (DEFAULT_TABLE table=default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.775679 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fbe09b4078c4eebb52d7d00bc87371c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:27.776790 9834 raft_consensus.cc:2468] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
W20260504 13:56:27.783547 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 451633us
W20260504 13:56:27.786051 9768 leader_election.cc:343] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: bd8cd1caaeb249e3bafee7f1f0d4df2f
I20260504 13:56:27.786535 9768 leader_election.cc:304] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:27.790798 10571 raft_consensus.cc:493] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:27.791252 10571 raft_consensus.cc:515] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.791880 9680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fbe09b4078c4eebb52d7d00bc87371c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:27.792447 9680 raft_consensus.cc:2468] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:27.792981 10571 leader_election.cc:290] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:27.793268 9620 leader_election.cc:304] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:27.793804 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f3905bfb30e4b03a7fcf0a2f57e2117" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:27.794734 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bae90898739b4303af0608e39e8c0367" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:27.795074 9473 consensus_peers.cc:597] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 930078a10afa4dd987ad021cf9609514. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:27.796391 10571 raft_consensus.cc:2804] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260504 13:56:27.796604 9618 leader_election.cc:343] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:27.796785 10571 raft_consensus.cc:493] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:27.797102 10571 raft_consensus.cc:3060] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:27.794476 9681 raft_consensus.cc:2393] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
W20260504 13:56:27.778708 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 7c9030802900499a90c4dbaf5c6936a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c9030802900499a90c4dbaf5c6936a4 already in progress: creating tablet
I20260504 13:56:27.800252 9655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c8e9ba29a1466082a3e8bedec87b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.800557 9932 tablet_bootstrap.cc:492] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:27.801661 9338 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]). Expected version 2 got 0
I20260504 13:56:27.805868 9339 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: default.Table_dee1335eff89438e9c68835a4c258031 [id=1714df2b31ff4865b6b357500d121a26] alter complete (version 1)
I20260504 13:56:27.810228 10571 raft_consensus.cc:515] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.810703 9932 tablet_bootstrap.cc:654] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:27.814534 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99e819494171485e9af8bc8ec33a6ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.816977 9656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ddf890c17964e7f90409ebe3c4034ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.819984 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc2f95dadd544f57bd6d1467d30e954d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.820202 10683 raft_consensus.cc:493] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:27.820626 10683 raft_consensus.cc:515] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.822360 10683 leader_election.cc:290] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:27.823712 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41d41f8ba6dc4bb18116cb2b2dfabff0" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:27.824827 9618 leader_election.cc:343] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:27.832773 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fbe09b4078c4eebb52d7d00bc87371c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:27.833293 9537 raft_consensus.cc:3060] T 1fbe09b4078c4eebb52d7d00bc87371c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:27.840293 9537 raft_consensus.cc:2468] T 1fbe09b4078c4eebb52d7d00bc87371c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:27.841392 9618 leader_election.cc:304] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:27.842152 10683 raft_consensus.cc:2804] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:27.842628 10683 raft_consensus.cc:697] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:27.843456 10683 consensus_queue.cc:237] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.844667 9336 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f] alter complete (version 1)
I20260504 13:56:27.845705 9340 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: default.Table_11d8ab61a6044154bb599875a0bc9004 [id=73e12524eb9e416782bf9f7fb58c93bd] alter complete (version 1)
I20260504 13:56:27.853808 9512 tablet_service.cc:1511] Processing CreateTablet for tablet da61c91e10264fa49fe397d9a9b014bb (DEFAULT_TABLE table=default.Table_999e37d0dab54ca68c0fa66f74582109 [id=e37fd7e36e604a60832acf303f0ecb7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.854835 9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da61c91e10264fa49fe397d9a9b014bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.860971 9659 tablet_service.cc:1511] Processing CreateTablet for tablet f71f444ce9214e3f98eff6ac8ca7dde7 (DEFAULT_TABLE table=default.Table_df8874a20e834872a12300a4f5628273 [id=7fc0413d79184a2dbe5c470b8506b4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.862193 9659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f71f444ce9214e3f98eff6ac8ca7dde7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.863391 10013 tablet_bootstrap.cc:492] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:27.863694 10571 leader_election.cc:290] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:27.864245 10013 ts_tablet_manager.cc:1403] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.201s user 0.007s sys 0.011s
W20260504 13:56:27.865842 9330 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet de6714ee86d64112baf13dd1b2b87582 (table default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet de6714ee86d64112baf13dd1b2b87582 consensus state has no leader
I20260504 13:56:27.867229 10013 raft_consensus.cc:359] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.867455 9657 tablet_service.cc:1511] Processing CreateTablet for tablet da61c91e10264fa49fe397d9a9b014bb (DEFAULT_TABLE table=default.Table_999e37d0dab54ca68c0fa66f74582109 [id=e37fd7e36e604a60832acf303f0ecb7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.867836 10013 raft_consensus.cc:385] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:27.868110 10013 raft_consensus.cc:740] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:27.868546 9657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da61c91e10264fa49fe397d9a9b014bb. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:27.882577 9344 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:55533 (request call id 41) took 4414 ms (client timeout 1000 ms). Trace:
I20260504 13:56:27.887284 10013 consensus_queue.cc:260] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.892117 10683 consensus_queue.cc:1048] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:27.898053 10086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 13:56:27.900547 10086 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 13:56:27.903162 9517 tablet_service.cc:1511] Processing CreateTablet for tablet f71f444ce9214e3f98eff6ac8ca7dde7 (DEFAULT_TABLE table=default.Table_df8874a20e834872a12300a4f5628273 [id=7fc0413d79184a2dbe5c470b8506b4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.854162 9767 leader_election.cc:304] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:27.907657 10013 ts_tablet_manager.cc:1434] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s
W20260504 13:56:27.883654 9344 rpcz_store.cc:269] 0504 13:56:23.468495 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:23.468612 (+ 117us) service_pool.cc:224] Handling call
0504 13:56:27.882513 (+4413901us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:23.803945 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:23.804334 (+ 389us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:23.804356 (+ 22us) write_op.cc:435] Acquired schema lock
0504 13:56:23.804367 (+ 11us) tablet.cc:662] Decoding operations
0504 13:56:23.804675 (+ 308us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:23.804715 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:23.804735 (+ 20us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:23.804882 (+ 147us) tablet.cc:701] Row locks acquired
0504 13:56:23.804896 (+ 14us) write_op.cc:260] PREPARE: finished
0504 13:56:23.805004 (+ 108us) write_op.cc:270] Start()
0504 13:56:23.805077 (+ 73us) write_op.cc:276] Timestamp: P: 1777902983804936 usec, L: 0
0504 13:56:23.805087 (+ 10us) op_driver.cc:348] REPLICATION: starting
0504 13:56:23.805534 (+ 447us) log.cc:844] Serialized 533 byte log entry
0504 13:56:23.817120 (+ 11586us) op_driver.cc:464] REPLICATION: finished
0504 13:56:23.817894 (+ 774us) write_op.cc:301] APPLY: starting
0504 13:56:23.817952 (+ 58us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:23.818032 (+ 80us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:23.818155 (+ 123us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:23.818319 (+ 164us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:23.820499 (+ 2180us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:23.820546 (+ 47us) write_op.cc:312] APPLY: finished
0504 13:56:23.820722 (+ 176us) log.cc:844] Serialized 23 byte log entry
0504 13:56:23.821348 (+ 626us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:23.821412 (+ 64us) write_op.cc:454] Released schema lock
0504 13:56:23.821442 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:23.821661 (+ 219us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:23.821689 (+ 28us) write_op.cc:454] Released schema lock
Related trace 'op':
0504 13:56:24.161935 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:24.162312 (+ 377us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:24.162338 (+ 26us) write_op.cc:435] Acquired schema lock
0504 13:56:24.162349 (+ 11us) tablet.cc:662] Decoding operations
0504 13:56:24.162590 (+ 241us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:24.162629 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:24.162652 (+ 23us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:24.162790 (+ 138us) tablet.cc:701] Row locks acquired
0504 13:56:24.162802 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:24.162901 (+ 99us) write_op.cc:270] Start()
0504 13:56:24.162977 (+ 76us) write_op.cc:276] Timestamp: P: 1777902984162849 usec, L: 0
0504 13:56:24.162992 (+ 15us) op_driver.cc:348] REPLICATION: starting
0504 13:56:24.163507 (+ 515us) log.cc:844] Serialized 356 byte log entry
0504 13:56:24.174608 (+ 11101us) op_driver.cc:464] REPLICATION: finished
0504 13:56:24.174926 (+ 318us) write_op.cc:301] APPLY: starting
0504 13:56:24.174980 (+ 54us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:24.175063 (+ 83us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:24.175157 (+ 94us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:24.175328 (+ 171us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:24.175455 (+ 127us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:24.175494 (+ 39us) write_op.cc:312] APPLY: finished
0504 13:56:24.175645 (+ 151us) log.cc:844] Serialized 23 byte log entry
0504 13:56:24.175777 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:24.175841 (+ 64us) write_op.cc:454] Released schema lock
0504 13:56:24.175874 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:24.176088 (+ 214us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:24.176115 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3335,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":597,"apply.run_cpu_time_us":1274,"apply.run_wall_time_us":3759,"num_ops":1,"prepare.queue_time_us":10190,"prepare.run_cpu_time_us":1805,"prepare.run_wall_time_us":1852,"raft.queue_time_us":151,"raft.run_cpu_time_us":606,"raft.run_wall_time_us":886,"replication_time_us":11948}],["op",{"apply.queue_time_us":169,"apply.run_cpu_time_us":1122,"apply.run_wall_time_us":1122,"num_ops":1,"prepare.queue_time_us":215,"prepare.run_cpu_time_us":1962,"prepare.run_wall_time_us":1962,"raft.queue_time_us":561,"raft.run_cpu_time_us":630,"raft.run_wall_time_us":637,"replication_time_us":11542,"spinlock_wait_cycles":103296}]]}
W20260504 13:56:27.906315 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 52227.8us
I20260504 13:56:27.920650 10699 raft_consensus.cc:2804] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:27.921079 10699 raft_consensus.cc:493] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:27.921386 10699 raft_consensus.cc:3060] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:27.923282 9932 tablet_bootstrap.cc:492] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:27.923668 9932 ts_tablet_manager.cc:1403] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.123s user 0.006s sys 0.004s
I20260504 13:56:27.923123 9507 tablet_service.cc:1511] Processing CreateTablet for tablet 36c8e9ba29a1466082a3e8bedec87b91 (DEFAULT_TABLE table=default.Table_1665799e268a496d9ffc5568fa34a763 [id=403cbbc8a1304e5695a89d3697932c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:27.932636 9932 raft_consensus.cc:359] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:27.933224 9932 raft_consensus.cc:385] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:27.933450 9932 raft_consensus.cc:740] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:27.923125 10697 raft_consensus.cc:2749] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260504 13:56:27.935127 9824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bae90898739b4303af0608e39e8c0367" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:27.946646 10086 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260504 13:56:27.926762 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 139932us
I20260504 13:56:27.943127 9932 consensus_queue.cc:260] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:27.955314 9620 leader_election.cc:343] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:27.955672 9620 leader_election.cc:304] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:27.956561 10571 raft_consensus.cc:2749] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:27.964927 9932 ts_tablet_manager.cc:1434] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.035s user 0.004s sys 0.002s
I20260504 13:56:27.977043 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f71f444ce9214e3f98eff6ac8ca7dde7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:27.985010 10391 ts_tablet_manager.cc:1929] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:27.985371 10391 log.cc:1199] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/030b35c3409143108c34ab1f0ee1b96d
I20260504 13:56:27.985728 9932 tablet_bootstrap.cc:492] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:27.986078 10391 ts_tablet_manager.cc:1950] T 030b35c3409143108c34ab1f0ee1b96d P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:28.001207 9657 tablet_service.cc:1511] Processing CreateTablet for tablet 21d6358b451d4b8d87a22e30e0780f44 (DEFAULT_TABLE table=default.Table_5bfac12ab2f84c06b1f992848d8f5041 [id=c2f8dc1bfbc14f8a8c093490ff05a18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:28.001205 9509 tablet_service.cc:1511] Processing CreateTablet for tablet 21d6358b451d4b8d87a22e30e0780f44 (DEFAULT_TABLE table=default.Table_5bfac12ab2f84c06b1f992848d8f5041 [id=c2f8dc1bfbc14f8a8c093490ff05a18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:28.002279 9657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21d6358b451d4b8d87a22e30e0780f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.003412 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fbe09b4078c4eebb52d7d00bc87371c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:28.009370 10683 raft_consensus.cc:493] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:28.009829 10683 raft_consensus.cc:515] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.011763 10683 leader_election.cc:290] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:28.013787 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68e92cc3da9b4bcfadb6fa26245f1c99" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:28.015028 9618 leader_election.cc:343] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:28.018098 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 47908.8us
I20260504 13:56:28.019819 9932 tablet_bootstrap.cc:654] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:28.019917 10726 consensus_queue.cc:1048] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:28.035298 10013 tablet_bootstrap.cc:492] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:28.037773 10700 raft_consensus.cc:493] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5beceb6317444247b31b44a6581707c6)
I20260504 13:56:28.039862 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60d3cc59ffe947fbb57b7d3301ccbac1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:28.042481 9932 tablet_bootstrap.cc:492] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:28.042840 9932 ts_tablet_manager.cc:1403] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.005s
I20260504 13:56:28.043828 10705 consensus_queue.cc:1048] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260504 13:56:28.044806 9337 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:55533 (request call id 42) took 3450 ms (client timeout 1000 ms). Trace:
W20260504 13:56:28.045200 9337 rpcz_store.cc:269] 0504 13:56:24.594256 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:24.612074 (+ 17818us) service_pool.cc:224] Handling call
0504 13:56:24.654060 (+ 41986us) spinlock_profiling.cc:243] Waited 39 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:28.044625 (+3390565us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21059,"reported_tablets":17,"spinlock_wait_cycles":89735936}
I20260504 13:56:28.046761 9507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c8e9ba29a1466082a3e8bedec87b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.045334 9932 raft_consensus.cc:359] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.047076 9932 raft_consensus.cc:385] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:28.047359 9932 raft_consensus.cc:740] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:28.048086 9932 consensus_queue.cc:260] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:28.050328 9339 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:38691 (request call id 40) took 3234 ms (client timeout 1000 ms). Trace:
W20260504 13:56:28.052515 10712 log.cc:927] Time spent T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260504 13:56:27.918136 9340 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: default.Table_72e3a254561b4cae927e6b6bcacd87b6 [id=f227576f105c4b1fbd459b3638ce684b] alter complete (version 2)
I20260504 13:56:28.054997 9509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21d6358b451d4b8d87a22e30e0780f44. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:28.055245 9620 leader_election.cc:343] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:28.055668 9620 leader_election.cc:304] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:28.056552 10683 raft_consensus.cc:2749] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:28.063910 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68e92cc3da9b4bcfadb6fa26245f1c99" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:28.065317 9666 tablet_service.cc:1558] Processing DeleteTablet for tablet 030b35c3409143108c34ab1f0ee1b96d with delete_type TABLET_DATA_DELETED (replaced by tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:28.068075 9666 tablet_service.cc:1558] Processing DeleteTablet for tablet 930078a10afa4dd987ad021cf9609514 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:28.070477 9656 tablet_service.cc:1558] Processing DeleteTablet for tablet 1501455a0af348e38746d100987afdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:28.072531 9666 tablet_service.cc:1558] Processing DeleteTablet for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 88619eaa917d4c33a910fc1347ab126e) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:28.073060 9665 tablet_service.cc:1558] Processing DeleteTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:16 UTC) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:28.073853 9342 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:38691 (request call id 39) took 4399 ms (client timeout 1000 ms). Trace:
W20260504 13:56:28.074050 9620 leader_election.cc:343] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:28.074396 9620 leader_election.cc:304] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:28.076838 10683 raft_consensus.cc:2749] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:28.077800 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 030b35c3409143108c34ab1f0ee1b96d because the tablet was not found. No further retry: Not found: Tablet not found: 030b35c3409143108c34ab1f0ee1b96d
I20260504 13:56:28.078357 9657 tablet_service.cc:1558] Processing DeleteTablet for tablet 48b193e958b4414aad29dd2bcee1a61b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:28.078730 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 930078a10afa4dd987ad021cf9609514 because the tablet was not found. No further retry: Not found: Tablet not found: 930078a10afa4dd987ad021cf9609514
W20260504 13:56:28.079300 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 1501455a0af348e38746d100987afdd0 because the tablet was not found. No further retry: Not found: Tablet not found: 1501455a0af348e38746d100987afdd0
W20260504 13:56:28.079847 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 because the tablet was not found. No further retry: Not found: Tablet not found: 9ed4f5ea230a413a8dcdbb661ce173b6
W20260504 13:56:28.080371 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f because the tablet was not found. No further retry: Not found: Tablet not found: bd8cd1caaeb249e3bafee7f1f0d4df2f
W20260504 13:56:28.080965 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 48b193e958b4414aad29dd2bcee1a61b because the tablet was not found. No further retry: Not found: Tablet not found: 48b193e958b4414aad29dd2bcee1a61b
I20260504 13:56:28.081066 10013 tablet_bootstrap.cc:654] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:28.074179 9342 rpcz_store.cc:269] 0504 13:56:23.674472 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:23.688090 (+ 13618us) service_pool.cc:224] Handling call
0504 13:56:23.743777 (+ 55687us) spinlock_profiling.cc:243] Waited 43 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:24.020421 (+276644us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:24.020445 (+ 24us) write_op.cc:454] Released schema lock
0504 13:56:28.073660 (+4053215us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:24.003296 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:24.003672 (+ 376us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:24.003697 (+ 25us) write_op.cc:435] Acquired schema lock
0504 13:56:24.003709 (+ 12us) tablet.cc:662] Decoding operations
0504 13:56:24.003958 (+ 249us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:24.003996 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:24.004017 (+ 21us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:24.004155 (+ 138us) tablet.cc:701] Row locks acquired
0504 13:56:24.004167 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:24.004268 (+ 101us) write_op.cc:270] Start()
0504 13:56:24.004347 (+ 79us) write_op.cc:276] Timestamp: P: 1777902984004225 usec, L: 0
0504 13:56:24.004363 (+ 16us) op_driver.cc:348] REPLICATION: starting
0504 13:56:24.004870 (+ 507us) log.cc:844] Serialized 533 byte log entry
0504 13:56:24.015755 (+ 10885us) op_driver.cc:464] REPLICATION: finished
0504 13:56:24.016162 (+ 407us) write_op.cc:301] APPLY: starting
0504 13:56:24.016215 (+ 53us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:24.016299 (+ 84us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:24.016379 (+ 80us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:24.016546 (+ 167us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:24.016666 (+ 120us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:24.016705 (+ 39us) write_op.cc:312] APPLY: finished
0504 13:56:24.016862 (+ 157us) log.cc:844] Serialized 23 byte log entry
0504 13:56:24.016981 (+ 119us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:24.017033 (+ 52us) write_op.cc:454] Released schema lock
0504 13:56:24.017061 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":70875,"reported_tablets":25,"spinlock_wait_cycles":98811136,"child_traces":[["op",{"apply.queue_time_us":263,"apply.run_cpu_time_us":1052,"apply.run_wall_time_us":1051,"num_ops":1,"prepare.queue_time_us":61545,"prepare.run_cpu_time_us":1869,"prepare.run_wall_time_us":1869,"raft.queue_time_us":2470,"raft.run_cpu_time_us":639,"raft.run_wall_time_us":639,"replication_time_us":11284}]]}
I20260504 13:56:28.084211 10705 consensus_queue.cc:1048] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:28.094830 9343 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:38691 (request call id 38) took 5466 ms (client timeout 1000 ms). Trace:
I20260504 13:56:28.096913 10697 consensus_queue.cc:1048] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:28.038255 10700 raft_consensus.cc:515] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:28.101833 10705 consensus_queue.cc:1048] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:28.105335 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 159) took 5070 ms. Trace:
I20260504 13:56:28.102304 10700 leader_election.cc:290] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:28.105741 10391 rpcz_store.cc:276] 0504 13:56:23.034884 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:23.036641 (+ 1757us) service_pool.cc:224] Handling call
0504 13:56:27.486562 (+4449921us) ts_tablet_manager.cc:1151] Deleting tablet 030b35c3409143108c34ab1f0ee1b96d
0504 13:56:27.486609 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:27.679844 (+193235us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:27.984902 (+305058us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:28.105266 (+120364us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4425837}
I20260504 13:56:28.109644 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41d41f8ba6dc4bb18116cb2b2dfabff0" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:28.110927 9932 ts_tablet_manager.cc:1434] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.068s user 0.003s sys 0.003s
W20260504 13:56:28.112540 9620 leader_election.cc:343] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:28.113248 9620 leader_election.cc:304] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:28.114004 10683 raft_consensus.cc:2749] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:28.114251 9341 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:55533 (request call id 44) took 1383 ms (client timeout 1000 ms). Trace:
W20260504 13:56:28.114639 9341 rpcz_store.cc:269] 0504 13:56:26.730329 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.104158 (+373829us) service_pool.cc:224] Handling call
0504 13:56:27.150287 (+ 46129us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:27.560733 (+410446us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:27.560757 (+ 24us) write_op.cc:454] Released schema lock
0504 13:56:28.113699 (+552942us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:27.542170 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:27.542720 (+ 550us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:27.542758 (+ 38us) write_op.cc:435] Acquired schema lock
0504 13:56:27.542781 (+ 23us) tablet.cc:662] Decoding operations
0504 13:56:27.543232 (+ 451us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:27.543294 (+ 62us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:27.543330 (+ 36us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:27.543675 (+ 345us) tablet.cc:701] Row locks acquired
0504 13:56:27.543698 (+ 23us) write_op.cc:260] PREPARE: finished
0504 13:56:27.543857 (+ 159us) write_op.cc:270] Start()
0504 13:56:27.543963 (+ 106us) write_op.cc:276] Timestamp: P: 1777902987543789 usec, L: 0
0504 13:56:27.543989 (+ 26us) op_driver.cc:348] REPLICATION: starting
0504 13:56:27.544634 (+ 645us) log.cc:844] Serialized 913 byte log entry
0504 13:56:27.555819 (+ 11185us) op_driver.cc:464] REPLICATION: finished
0504 13:56:27.556058 (+ 239us) write_op.cc:301] APPLY: starting
0504 13:56:27.556109 (+ 51us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:27.556213 (+ 104us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:27.556309 (+ 96us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:27.556549 (+ 240us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:27.556682 (+ 133us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:27.556721 (+ 39us) write_op.cc:312] APPLY: finished
0504 13:56:27.556894 (+ 173us) log.cc:844] Serialized 29 byte log entry
0504 13:56:27.557026 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:27.557101 (+ 75us) write_op.cc:454] Released schema lock
0504 13:56:27.557130 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":44433,"reported_tablets":26,"spinlock_wait_cycles":32296832,"child_traces":[["op",{"apply.queue_time_us":74,"apply.run_cpu_time_us":1265,"apply.run_wall_time_us":1265,"num_ops":2,"prepare.queue_time_us":13102,"prepare.run_cpu_time_us":2889,"prepare.run_wall_time_us":2889,"raft.queue_time_us":3071,"raft.run_cpu_time_us":833,"raft.run_wall_time_us":835,"replication_time_us":11714}]]}
I20260504 13:56:28.139788 9655 tablet_service.cc:1558] Processing DeleteTablet for tablet c34877a81d504f46a2f680c5d5ddb4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 359b0fa01a1648a7bef6f97960a7b9a8) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:28.140535 9932 tablet_bootstrap.cc:492] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:28.148133 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 60071.7us
I20260504 13:56:28.107116 10391 tablet_replica.cc:333] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:28.149338 10391 raft_consensus.cc:2243] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:28.150121 10391 pending_rounds.cc:70] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00: Trying to abort 1 pending ops.
I20260504 13:56:28.150352 10391 pending_rounds.cc:77] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290634453331968 op_type: NO_OP noop_request { }
I20260504 13:56:28.150080 9341 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_28052c777ee944b79b802d28b415c24e" } new_table_name: "default.Table_3bc454a7435a4553be5a9bbcfec24eb3" modify_external_catalogs: true
I20260504 13:56:28.150661 10391 raft_consensus.cc:2889] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:28.120857 9337 master_service.cc:954] ReplaceTablet: received request to replace tablet 48b193e958b4414aad29dd2bcee1a61b from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:28.085845 9342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_6b0124eccb02454b9195b8ddaae5bd38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:28.152477 9342 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b0124eccb02454b9195b8ddaae5bd38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:28.036154 9336 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:55533 (request call id 43) took 2430 ms (client timeout 1000 ms). Trace:
W20260504 13:56:28.153059 9336 rpcz_store.cc:269] 0504 13:56:25.605283 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:25.655487 (+ 50204us) service_pool.cc:224] Handling call
0504 13:56:27.837766 (+2182279us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:27.837791 (+ 25us) write_op.cc:454] Released schema lock
0504 13:56:28.035941 (+198150us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:27.817532 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:27.817999 (+ 467us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:27.818030 (+ 31us) write_op.cc:435] Acquired schema lock
0504 13:56:27.818047 (+ 17us) tablet.cc:662] Decoding operations
0504 13:56:27.818328 (+ 281us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:27.818368 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:27.818392 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:27.818557 (+ 165us) tablet.cc:701] Row locks acquired
0504 13:56:27.818572 (+ 15us) write_op.cc:260] PREPARE: finished
0504 13:56:27.818692 (+ 120us) write_op.cc:270] Start()
0504 13:56:27.818788 (+ 96us) write_op.cc:276] Timestamp: P: 1777902987818623 usec, L: 0
0504 13:56:27.818807 (+ 19us) op_driver.cc:348] REPLICATION: starting
0504 13:56:27.820317 (+ 1510us) log.cc:844] Serialized 356 byte log entry
0504 13:56:27.829832 (+ 9515us) op_driver.cc:464] REPLICATION: finished
0504 13:56:27.830145 (+ 313us) write_op.cc:301] APPLY: starting
0504 13:56:27.830198 (+ 53us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:27.830315 (+ 117us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:27.830400 (+ 85us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:27.830542 (+ 142us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:27.830673 (+ 131us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:27.830773 (+ 100us) write_op.cc:312] APPLY: finished
0504 13:56:27.830948 (+ 175us) log.cc:844] Serialized 23 byte log entry
0504 13:56:27.831084 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:27.831146 (+ 62us) write_op.cc:454] Released schema lock
0504 13:56:27.832619 (+ 1473us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":17001,"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":185,"apply.run_cpu_time_us":1244,"apply.run_wall_time_us":2653,"num_ops":1,"prepare.queue_time_us":89,"prepare.run_cpu_time_us":2595,"prepare.run_wall_time_us":3060,"raft.queue_time_us":1055,"raft.run_cpu_time_us":519,"raft.run_wall_time_us":520,"replication_time_us":10946}]]}
W20260504 13:56:28.095268 9343 rpcz_store.cc:269] 0504 13:56:22.628276 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:22.630483 (+ 2207us) service_pool.cc:224] Handling call
0504 13:56:22.638674 (+ 8191us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:28.094646 (+5455972us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:23.688820 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:23.689191 (+ 371us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:23.689214 (+ 23us) write_op.cc:435] Acquired schema lock
0504 13:56:23.689227 (+ 13us) tablet.cc:662] Decoding operations
0504 13:56:23.689498 (+ 271us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:23.689530 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:23.689552 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:23.689772 (+ 220us) tablet.cc:701] Row locks acquired
0504 13:56:23.689784 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:23.689887 (+ 103us) write_op.cc:270] Start()
0504 13:56:23.689958 (+ 71us) write_op.cc:276] Timestamp: P: 1777902983689831 usec, L: 0
0504 13:56:23.689972 (+ 14us) op_driver.cc:348] REPLICATION: starting
0504 13:56:23.743823 (+ 53851us) log.cc:844] Serialized 913 byte log entry
0504 13:56:23.744461 (+ 638us) spinlock_profiling.cc:243] Waited 53.3 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b801951 00007fc27b894208 00007fc27bd069f1 00007fc27bd061c8 00007fc27bd0ae61 00007fc27bd0ae19 00007fc27bd0ada9 00007fc27bd0ad71 00007fc27bd09f9d 00000000005b3bc4 00000000005b39f9
0504 13:56:23.749821 (+ 5360us) spinlock_profiling.cc:243] Waited 3.51 ms on lock 0x7b5400030aa0. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b801951 00007fc27b811522 00007fc27b7f228a 00007fc27b7f9133 00007fc27b7f90b9 00007fc27b7f9049 00007fc27b7f9011 00007fc27b7f81dd 00000000005b3bc4 00000000005b39f9 00007fc2747dbab5
0504 13:56:23.760856 (+ 11035us) op_driver.cc:464] REPLICATION: finished
0504 13:56:23.769854 (+ 8998us) write_op.cc:301] APPLY: starting
0504 13:56:23.769914 (+ 60us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:23.770016 (+ 102us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:23.770035 (+ 19us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:23.770263 (+ 228us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:23.770409 (+ 146us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:23.770455 (+ 46us) write_op.cc:312] APPLY: finished
0504 13:56:23.770620 (+ 165us) log.cc:844] Serialized 29 byte log entry
0504 13:56:23.770764 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:23.770830 (+ 66us) write_op.cc:454] Released schema lock
0504 13:56:23.770868 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:23.771101 (+ 233us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:23.771132 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":11570,"reported_tablets":24,"spinlock_wait_cycles":5595008,"child_traces":[["op",{"apply.queue_time_us":8870,"apply.run_cpu_time_us":1199,"apply.run_wall_time_us":1199,"num_ops":2,"prepare.queue_time_us":106,"prepare.run_cpu_time_us":3049,"prepare.run_wall_time_us":55804,"raft.queue_time_us":152,"raft.run_cpu_time_us":2185,"raft.run_wall_time_us":5618,"replication_time_us":70801,"spinlock_wait_cycles":130682624}]]}
I20260504 13:56:28.159137 9343 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_0d4595642a604b8f8f57d3eadbbacacb" } modify_external_catalogs: true
I20260504 13:56:28.166533 9659 tablet_service.cc:1558] Processing DeleteTablet for tablet 1501455a0af348e38746d100987afdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:28.167876 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet c34877a81d504f46a2f680c5d5ddb4f4 because the tablet was not found. No further retry: Not found: Tablet not found: c34877a81d504f46a2f680c5d5ddb4f4
I20260504 13:56:28.169757 9655 tablet_service.cc:1558] Processing DeleteTablet for tablet 48b193e958b4414aad29dd2bcee1a61b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:28.150991 10391 raft_consensus.cc:2272] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:28.170959 9336 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_0e2a1bfda0a945dc86c7e8e9bd28fa49" } new_table_name: "default.Table_59e33df553a4469b8fd50d46d7ce9f97" modify_external_catalogs: true
I20260504 13:56:28.172986 10391 ts_tablet_manager.cc:1916] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:28.179795 9931 tablet_bootstrap.cc:492] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:28.180209 9931 ts_tablet_manager.cc:1403] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 1.119s user 0.026s sys 0.013s
I20260504 13:56:28.182221 10013 tablet_bootstrap.cc:492] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:28.182605 10013 ts_tablet_manager.cc:1403] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.148s user 0.010s sys 0.000s
I20260504 13:56:28.182588 9931 raft_consensus.cc:359] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:28.185333 9335 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 35) took 3144 ms (client timeout 1000 ms). Trace:
W20260504 13:56:28.185680 9335 rpcz_store.cc:269] 0504 13:56:25.040803 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:25.321605 (+280802us) service_pool.cc:224] Handling call
0504 13:56:28.185179 (+2863574us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5790,"reported_tablets":29,"spinlock_wait_cycles":1678592}
W20260504 13:56:28.050652 9339 rpcz_store.cc:269] 0504 13:56:24.815573 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:25.078552 (+262979us) service_pool.cc:224] Handling call
0504 13:56:25.267729 (+189177us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:27.180004 (+1912275us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:27.180028 (+ 24us) write_op.cc:454] Released schema lock
0504 13:56:28.050161 (+870133us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:27.031779 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:27.032243 (+ 464us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:27.032268 (+ 25us) write_op.cc:435] Acquired schema lock
0504 13:56:27.032280 (+ 12us) tablet.cc:662] Decoding operations
0504 13:56:27.032560 (+ 280us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:27.032600 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:27.032644 (+ 44us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:27.032794 (+ 150us) tablet.cc:701] Row locks acquired
0504 13:56:27.032807 (+ 13us) write_op.cc:260] PREPARE: finished
0504 13:56:27.033505 (+ 698us) write_op.cc:270] Start()
0504 13:56:27.033595 (+ 90us) write_op.cc:276] Timestamp: P: 1777902987033465 usec, L: 0
0504 13:56:27.033613 (+ 18us) op_driver.cc:348] REPLICATION: starting
0504 13:56:27.034090 (+ 477us) log.cc:844] Serialized 533 byte log entry
0504 13:56:27.110403 (+ 76313us) op_driver.cc:464] REPLICATION: finished
0504 13:56:27.112309 (+ 1906us) write_op.cc:301] APPLY: starting
0504 13:56:27.112353 (+ 44us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:27.112439 (+ 86us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:27.112514 (+ 75us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:27.112666 (+ 152us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:27.112772 (+ 106us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:27.112814 (+ 42us) write_op.cc:312] APPLY: finished
0504 13:56:27.112957 (+ 143us) log.cc:844] Serialized 23 byte log entry
0504 13:56:27.113260 (+ 303us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:27.113313 (+ 53us) write_op.cc:454] Released schema lock
0504 13:56:27.113340 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0504 13:56:27.158555 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:27.159010 (+ 455us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:27.159036 (+ 26us) write_op.cc:435] Acquired schema lock
0504 13:56:27.159055 (+ 19us) tablet.cc:662] Decoding operations
0504 13:56:27.159346 (+ 291us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:27.159388 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:27.159417 (+ 29us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:27.159562 (+ 145us) tablet.cc:701] Row locks acquired
0504 13:56:27.159578 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:27.159692 (+ 114us) write_op.cc:270] Start()
0504 13:56:27.159773 (+ 81us) write_op.cc:276] Timestamp: P: 1777902987159644 usec, L: 0
0504 13:56:27.159793 (+ 20us) op_driver.cc:348] REPLICATION: starting
0504 13:56:27.160350 (+ 557us) log.cc:844] Serialized 356 byte log entry
0504 13:56:27.167535 (+ 7185us) op_driver.cc:464] REPLICATION: finished
0504 13:56:27.169176 (+ 1641us) write_op.cc:301] APPLY: starting
0504 13:56:27.169225 (+ 49us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:27.169305 (+ 80us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:27.169398 (+ 93us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:27.169554 (+ 156us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:27.169668 (+ 114us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:27.169704 (+ 36us) write_op.cc:312] APPLY: finished
0504 13:56:27.169864 (+ 160us) log.cc:844] Serialized 23 byte log entry
0504 13:56:27.170415 (+ 551us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:27.170469 (+ 54us) write_op.cc:454] Released schema lock
0504 13:56:27.170498 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":8332,"reported_tablets":25,"spinlock_wait_cycles":425078016,"child_traces":[["op",{"apply.queue_time_us":1784,"apply.run_cpu_time_us":1049,"apply.run_wall_time_us":1198,"num_ops":1,"prepare.queue_time_us":4954,"prepare.run_cpu_time_us":2463,"prepare.run_wall_time_us":2542,"replication_time_us":76707,"thread_start_us":38577,"threads_started":1}],["op",{"apply.queue_time_us":1494,"apply.run_cpu_time_us":1092,"apply.run_wall_time_us":1504,"num_ops":1,"prepare.queue_time_us":82,"prepare.run_cpu_time_us":2056,"prepare.run_wall_time_us":2057,"raft.queue_time_us":169,"raft.run_cpu_time_us":486,"raft.run_wall_time_us":485,"replication_time_us":7668}]]}
I20260504 13:56:28.195834 9337 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:35768 (request call id 258) took 2493 ms. Trace:
I20260504 13:56:28.196214 9337 rpcz_store.cc:276] 0504 13:56:25.702162 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.120693 (+2418531us) service_pool.cc:224] Handling call
0504 13:56:28.195773 (+ 75080us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:28.197595 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_512ae07c2df74cb99bb77006d985f2da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:28.199105 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_512ae07c2df74cb99bb77006d985f2da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:28.126725 9338 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:38691 (request call id 42) took 1041 ms (client timeout 1000 ms). Trace:
I20260504 13:56:28.200209 9657 tablet_service.cc:1558] Processing DeleteTablet for tablet baf07a970a7c4a35b3c849b06d6c88d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:24 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:28.200711 10013 raft_consensus.cc:359] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:28.062906 9340 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 34) took 4504 ms (client timeout 1000 ms). Trace:
I20260504 13:56:28.201365 10013 raft_consensus.cc:385] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:28.201650 10013 raft_consensus.cc:740] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:28.201684 9657 tablet_service.cc:1558] Processing DeleteTablet for tablet 930078a10afa4dd987ad021cf9609514 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:28.205721 9932 tablet_bootstrap.cc:654] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:28.207719 9657 tablet_service.cc:1558] Processing DeleteTablet for tablet 030b35c3409143108c34ab1f0ee1b96d with delete_type TABLET_DATA_DELETED (replaced by tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:28.212872 9657 tablet_service.cc:1558] Processing DeleteTablet for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 88619eaa917d4c33a910fc1347ab126e) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:28.216396 10013 consensus_queue.cc:260] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.218825 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 030b35c3409143108c34ab1f0ee1b96d (table default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]) successfully deleted
I20260504 13:56:28.220330 9512 tablet_service.cc:1558] Processing DeleteTablet for tablet 65052a5c28c147a385144317e21eea41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:13 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:28.220470 9657 tablet_service.cc:1558] Processing DeleteTablet for tablet 604f1ba5f71041a7877daff8972a994a with delete_type TABLET_DATA_DELETED (replaced by tablet 2ddf890c17964e7f90409ebe3c4034ef) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:28.222472 10013 ts_tablet_manager.cc:1434] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.040s user 0.007s sys 0.002s
W20260504 13:56:28.225075 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 1501455a0af348e38746d100987afdd0 because the tablet was not found. No further retry: Not found: Tablet not found: 1501455a0af348e38746d100987afdd0
I20260504 13:56:28.223860 9343 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_0d4595642a604b8f8f57d3eadbbacacb" } modify_external_catalogs: true
W20260504 13:56:28.225754 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 48b193e958b4414aad29dd2bcee1a61b because the tablet was not found. No further retry: Not found: Tablet not found: 48b193e958b4414aad29dd2bcee1a61b
W20260504 13:56:28.226341 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet baf07a970a7c4a35b3c849b06d6c88d0 because the tablet was not found. No further retry: Not found: Tablet not found: baf07a970a7c4a35b3c849b06d6c88d0
W20260504 13:56:28.226962 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 930078a10afa4dd987ad021cf9609514 because the tablet was not found. No further retry: Not found: Tablet not found: 930078a10afa4dd987ad021cf9609514
W20260504 13:56:28.227586 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 030b35c3409143108c34ab1f0ee1b96d because the tablet was not found. No further retry: Not found: Tablet not found: 030b35c3409143108c34ab1f0ee1b96d
W20260504 13:56:28.228125 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 because the tablet was not found. No further retry: Not found: Tablet not found: 9ed4f5ea230a413a8dcdbb661ce173b6
W20260504 13:56:28.228750 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 604f1ba5f71041a7877daff8972a994a because the tablet was not found. No further retry: Not found: Tablet not found: 604f1ba5f71041a7877daff8972a994a
I20260504 13:56:28.233954 10683 consensus_queue.cc:1048] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:28.244621 10013 tablet_bootstrap.cc:492] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:28.244831 9507 tablet_service.cc:1558] Processing DeleteTablet for tablet 50e129980cd84f6ca22cbf2addb06230 with delete_type TABLET_DATA_DELETED (replaced by tablet c47b6e3b4d0c45c28384f7aae6b749de) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:28.246640 9507 tablet_service.cc:1558] Processing DeleteTablet for tablet 48b193e958b4414aad29dd2bcee1a61b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:28.257263 9509 tablet_service.cc:1558] Processing DeleteTablet for tablet 3c43ff1d759a40d68344f25f066416ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:17 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:28.259696 10013 tablet_bootstrap.cc:654] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:28.264803 9509 tablet_service.cc:1558] Processing DeleteTablet for tablet 07a428049fdb4a68aae729400e2d9674 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:14 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:28.276321 9932 tablet_bootstrap.cc:492] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:28.276887 9932 ts_tablet_manager.cc:1403] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.137s user 0.003s sys 0.008s
I20260504 13:56:28.281100 9932 raft_consensus.cc:359] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.283524 9932 raft_consensus.cc:385] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:28.283828 9932 raft_consensus.cc:740] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:28.285418 9932 consensus_queue.cc:260] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:28.199664 9338 rpcz_store.cc:269] 0504 13:56:27.085372 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.130583 (+ 45211us) service_pool.cc:224] Handling call
0504 13:56:27.886908 (+756325us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:27.886940 (+ 32us) write_op.cc:454] Released schema lock
0504 13:56:28.126544 (+239604us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:27.868130 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:27.868569 (+ 439us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:27.868594 (+ 25us) write_op.cc:435] Acquired schema lock
0504 13:56:27.868611 (+ 17us) tablet.cc:662] Decoding operations
0504 13:56:27.869061 (+ 450us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:27.869105 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:27.869130 (+ 25us) tablet.cc:685] Acquiring locks for 3 operations
0504 13:56:27.869414 (+ 284us) tablet.cc:701] Row locks acquired
0504 13:56:27.869430 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:27.869550 (+ 120us) write_op.cc:270] Start()
0504 13:56:27.869645 (+ 95us) write_op.cc:276] Timestamp: P: 1777902987869481 usec, L: 0
0504 13:56:27.869665 (+ 20us) op_driver.cc:348] REPLICATION: starting
0504 13:56:27.870222 (+ 557us) log.cc:844] Serialized 1293 byte log entry
0504 13:56:27.880068 (+ 9846us) op_driver.cc:464] REPLICATION: finished
0504 13:56:27.880388 (+ 320us) write_op.cc:301] APPLY: starting
0504 13:56:27.880439 (+ 51us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:27.880547 (+ 108us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:27.880623 (+ 76us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:27.880910 (+ 287us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:27.881034 (+ 124us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0504 13:56:27.881071 (+ 37us) write_op.cc:312] APPLY: finished
0504 13:56:27.881238 (+ 167us) log.cc:844] Serialized 35 byte log entry
0504 13:56:27.881375 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:27.881441 (+ 66us) write_op.cc:454] Released schema lock
0504 13:56:27.881468 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":19715,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":187,"apply.run_cpu_time_us":1256,"apply.run_wall_time_us":1256,"num_ops":3,"prepare.queue_time_us":20838,"prepare.run_cpu_time_us":2563,"prepare.run_wall_time_us":5229,"raft.queue_time_us":76,"raft.run_cpu_time_us":601,"raft.run_wall_time_us":601,"replication_time_us":10318,"spinlock_wait_cycles":642304}]]}
I20260504 13:56:28.289265 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_861ff05bb9e144ea85a3fb7d75f68749"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:28.290664 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_861ff05bb9e144ea85a3fb7d75f68749 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:28.291932 9509 tablet_service.cc:1558] Processing DeleteTablet for tablet c34877a81d504f46a2f680c5d5ddb4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 359b0fa01a1648a7bef6f97960a7b9a8) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:28.293089 9509 tablet_service.cc:1558] Processing DeleteTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:16 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:28.294171 9509 tablet_service.cc:1558] Processing DeleteTablet for tablet afad9329cf7745ba8168b2a93f998295 with delete_type TABLET_DATA_DELETED (replaced by tablet af5c5a6cae4e4b6ea3eece2e21a587ec) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:28.295361 9509 tablet_service.cc:1558] Processing DeleteTablet for tablet 70caac2e256d419dbc2e70cb5dfffe97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:18 UTC) from {username='slave'} at 127.0.0.1:52642
W20260504 13:56:28.201463 9340 rpcz_store.cc:269] 0504 13:56:23.558038 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:23.684130 (+126092us) service_pool.cc:224] Handling call
0504 13:56:23.740683 (+ 56553us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:28.062730 (+4322047us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:24.165645 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:24.165981 (+ 336us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:24.166003 (+ 22us) write_op.cc:435] Acquired schema lock
0504 13:56:24.166014 (+ 11us) tablet.cc:662] Decoding operations
0504 13:56:24.166244 (+ 230us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:24.166273 (+ 29us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:24.166303 (+ 30us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:24.166481 (+ 178us) tablet.cc:701] Row locks acquired
0504 13:56:24.166499 (+ 18us) write_op.cc:260] PREPARE: finished
0504 13:56:24.166619 (+ 120us) write_op.cc:270] Start()
0504 13:56:24.166717 (+ 98us) write_op.cc:276] Timestamp: P: 1777902984166559 usec, L: 0
0504 13:56:24.166817 (+ 100us) op_driver.cc:348] REPLICATION: starting
0504 13:56:24.167298 (+ 481us) log.cc:844] Serialized 533 byte log entry
0504 13:56:24.184750 (+ 17452us) op_driver.cc:464] REPLICATION: finished
0504 13:56:24.185365 (+ 615us) write_op.cc:301] APPLY: starting
0504 13:56:24.185422 (+ 57us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:24.185501 (+ 79us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:24.185573 (+ 72us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:24.185706 (+ 133us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:24.185846 (+ 140us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:24.185885 (+ 39us) write_op.cc:312] APPLY: finished
0504 13:56:24.186047 (+ 162us) log.cc:844] Serialized 23 byte log entry
0504 13:56:24.186189 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:24.186238 (+ 49us) write_op.cc:454] Released schema lock
0504 13:56:24.186269 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:24.186466 (+ 197us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:24.186494 (+ 28us) write_op.cc:454] Released schema lock
Related trace 'op':
0504 13:56:24.331454 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:24.331842 (+ 388us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:24.331866 (+ 24us) write_op.cc:435] Acquired schema lock
0504 13:56:24.331877 (+ 11us) tablet.cc:662] Decoding operations
0504 13:56:24.332128 (+ 251us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:24.332163 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:24.332194 (+ 31us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:24.332333 (+ 139us) tablet.cc:701] Row locks acquired
0504 13:56:24.332346 (+ 13us) write_op.cc:260] PREPARE: finished
0504 13:56:24.332448 (+ 102us) write_op.cc:270] Start()
0504 13:56:24.332527 (+ 79us) write_op.cc:276] Timestamp: P: 1777902984332381 usec, L: 0
0504 13:56:24.332544 (+ 17us) op_driver.cc:348] REPLICATION: starting
0504 13:56:24.333011 (+ 467us) log.cc:844] Serialized 356 byte log entry
0504 13:56:24.940811 (+607800us) op_driver.cc:464] REPLICATION: finished
0504 13:56:24.941024 (+ 213us) write_op.cc:301] APPLY: starting
0504 13:56:24.941071 (+ 47us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:24.941159 (+ 88us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:24.941234 (+ 75us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:24.941391 (+ 157us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:24.941515 (+ 124us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:24.941553 (+ 38us) write_op.cc:312] APPLY: finished
0504 13:56:24.941716 (+ 163us) log.cc:844] Serialized 23 byte log entry
0504 13:56:24.941853 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:24.941908 (+ 55us) write_op.cc:454] Released schema lock
0504 13:56:24.941935 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0504 13:56:27.902087 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:27.902635 (+ 548us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:27.902662 (+ 27us) write_op.cc:435] Acquired schema lock
0504 13:56:27.902696 (+ 34us) tablet.cc:662] Decoding operations
0504 13:56:27.902972 (+ 276us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:27.903014 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:27.903038 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:27.903208 (+ 170us) tablet.cc:701] Row locks acquired
0504 13:56:27.903227 (+ 19us) write_op.cc:260] PREPARE: finished
0504 13:56:27.903346 (+ 119us) write_op.cc:270] Start()
0504 13:56:27.903438 (+ 92us) write_op.cc:276] Timestamp: P: 1777902987903276 usec, L: 0
0504 13:56:27.903452 (+ 14us) op_driver.cc:348] REPLICATION: starting
0504 13:56:27.903988 (+ 536us) log.cc:844] Serialized 356 byte log entry
0504 13:56:27.911037 (+ 7049us) op_driver.cc:464] REPLICATION: finished
0504 13:56:27.911303 (+ 266us) write_op.cc:301] APPLY: starting
0504 13:56:27.911350 (+ 47us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:27.911428 (+ 78us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:27.911501 (+ 73us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:27.911676 (+ 175us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:27.911797 (+ 121us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:27.911836 (+ 39us) write_op.cc:312] APPLY: finished
0504 13:56:27.911975 (+ 139us) log.cc:844] Serialized 23 byte log entry
0504 13:56:27.912563 (+ 588us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:27.912617 (+ 54us) write_op.cc:454] Released schema lock
0504 13:56:27.912647 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:27.912849 (+ 202us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:27.912871 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":139695,"reported_tablets":29,"spinlock_wait_cycles":58293632,"child_traces":[["op",{"apply.queue_time_us":469,"apply.run_cpu_time_us":1069,"apply.run_wall_time_us":1069,"num_ops":1,"prepare.queue_time_us":75,"prepare.run_cpu_time_us":1806,"prepare.run_wall_time_us":3185,"raft.queue_time_us":1527,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":15,"replication_time_us":17841}],["op",{"apply.queue_time_us":67,"apply.run_cpu_time_us":1143,"apply.run_wall_time_us":1143,"num_ops":1,"prepare.queue_time_us":69048,"prepare.run_cpu_time_us":1779,"prepare.run_wall_time_us":31269,"replication_time_us":608181}],["op",{"apply.queue_time_us":150,"apply.run_cpu_time_us":1516,"apply.run_wall_time_us":1515,"num_ops":1,"prepare.queue_time_us":21584,"prepare.run_cpu_time_us":2251,"prepare.run_wall_time_us":3089,"raft.queue_time_us":97,"raft.run_cpu_time_us":623,"raft.run_wall_time_us":622,"replication_time_us":7511}]]}
I20260504 13:56:28.302927 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_862ae682dfb44737a72e7c5df628a957"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:28.205260 9813 tablet_service.cc:1511] Processing CreateTablet for tablet 99e819494171485e9af8bc8ec33a6ff0 (DEFAULT_TABLE table=default.Table_89c38f9ba46e48fab397a7f3d5d1b805 [id=e4f6621892874f979d1c579d487cba44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:28.018453 10728 raft_consensus.cc:493] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:28.207975 9814 tablet_service.cc:1511] Processing CreateTablet for tablet 36c8e9ba29a1466082a3e8bedec87b91 (DEFAULT_TABLE table=default.Table_1665799e268a496d9ffc5568fa34a763 [id=403cbbc8a1304e5695a89d3697932c26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:28.214643 9809 tablet_service.cc:1511] Processing CreateTablet for tablet cc2f95dadd544f57bd6d1467d30e954d (DEFAULT_TABLE table=default.Table_2585508fc4214c8b82d2957644309ac2 [id=256b605e5a2342ed883ec30e3de0ffda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:28.215553 9807 tablet_service.cc:1511] Processing CreateTablet for tablet 2ddf890c17964e7f90409ebe3c4034ef (DEFAULT_TABLE table=default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]), partition=RANGE (key) PARTITION UNBOUNDED
W20260504 13:56:28.216812 9767 leader_election.cc:343] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Not found: Tablet not found: 030b35c3409143108c34ab1f0ee1b96d
I20260504 13:56:28.218923 9802 tablet_service.cc:1511] Processing CreateTablet for tablet da61c91e10264fa49fe397d9a9b014bb (DEFAULT_TABLE table=default.Table_999e37d0dab54ca68c0fa66f74582109 [id=e37fd7e36e604a60832acf303f0ecb7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:28.219430 9796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af5c5a6cae4e4b6ea3eece2e21a587ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.314277 9932 ts_tablet_manager.cc:1434] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.037s user 0.005s sys 0.001s
W20260504 13:56:28.212404 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 193595us
I20260504 13:56:28.221076 9810 tablet_service.cc:1511] Processing CreateTablet for tablet f71f444ce9214e3f98eff6ac8ca7dde7 (DEFAULT_TABLE table=default.Table_df8874a20e834872a12300a4f5628273 [id=7fc0413d79184a2dbe5c470b8506b4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:28.221925 9811 tablet_service.cc:1558] Processing DeleteTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:16 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:28.206987 10699 raft_consensus.cc:515] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.313156 9931 raft_consensus.cc:385] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:28.325102 9931 raft_consensus.cc:740] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:28.315121 9932 tablet_bootstrap.cc:492] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:28.327670 9798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.328478 9810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f71f444ce9214e3f98eff6ac8ca7dde7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.280617 9799 tablet_service.cc:1558] Processing DeleteTablet for tablet 1501455a0af348e38746d100987afdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:28.336710 9834 raft_consensus.cc:2393] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5beceb6317444247b31b44a6581707c6 in current term 1: Already voted for candidate d4051d2bf0bf410eb8acd107560df888 in this term.
I20260504 13:56:28.336997 9799 tablet_service.cc:1511] Processing CreateTablet for tablet 21d6358b451d4b8d87a22e30e0780f44 (DEFAULT_TABLE table=default.Table_5bfac12ab2f84c06b1f992848d8f5041 [id=c2f8dc1bfbc14f8a8c093490ff05a18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:28.281868 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet c34877a81d504f46a2f680c5d5ddb4f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 359b0fa01a1648a7bef6f97960a7b9a8) from {username='slave'} at 127.0.0.1:56380
W20260504 13:56:28.312619 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 95728.3us
I20260504 13:56:28.331856 9931 consensus_queue.cc:260] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.340801 10699 leader_election.cc:290] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:28.341221 9768 leader_election.cc:304] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:28.342957 9803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6103d953ace4cb8be3c580b3316f87c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.344033 9814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c8e9ba29a1466082a3e8bedec87b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.344550 9807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ddf890c17964e7f90409ebe3c4034ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.345122 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99e819494171485e9af8bc8ec33a6ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.345705 9809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc2f95dadd544f57bd6d1467d30e954d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.270733 10729 raft_consensus.cc:493] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:28.348340 9342 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35768 (request call id 257) took 2819 ms. Trace:
I20260504 13:56:28.349006 9932 tablet_bootstrap.cc:654] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:28.308586 10728 raft_consensus.cc:515] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.349603 9509 tablet_service.cc:1558] Processing DeleteTablet for tablet c34630d55f4a4e19b33ddf4c13995a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:19 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:28.357518 9342 rpcz_store.cc:276] 0504 13:56:25.528482 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.085333 (+2556851us) service_pool.cc:224] Handling call
0504 13:56:28.211098 (+125765us) catalog_manager.cc:2101] Acquired catalog manager lock
0504 13:56:28.248189 (+ 37091us) catalog_manager.cc:2147] Created new table and tablet info
0504 13:56:28.347760 (+ 99571us) catalog_manager.cc:2207] Wrote table and tablets to system table
0504 13:56:28.348036 (+ 276us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0504 13:56:28.348267 (+ 231us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:28.304568 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:28.305084 (+ 516us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:28.305110 (+ 26us) write_op.cc:435] Acquired schema lock
0504 13:56:28.305125 (+ 15us) tablet.cc:662] Decoding operations
0504 13:56:28.305404 (+ 279us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:28.305466 (+ 62us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:28.305491 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:28.305698 (+ 207us) tablet.cc:701] Row locks acquired
0504 13:56:28.305717 (+ 19us) write_op.cc:260] PREPARE: finished
0504 13:56:28.305840 (+ 123us) write_op.cc:270] Start()
0504 13:56:28.305926 (+ 86us) write_op.cc:276] Timestamp: P: 1777902988305745 usec, L: 0
0504 13:56:28.305943 (+ 17us) op_driver.cc:348] REPLICATION: starting
0504 13:56:28.306464 (+ 521us) log.cc:844] Serialized 433 byte log entry
0504 13:56:28.322742 (+ 16278us) op_driver.cc:464] REPLICATION: finished
0504 13:56:28.323461 (+ 719us) write_op.cc:301] APPLY: starting
0504 13:56:28.323511 (+ 50us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:28.323609 (+ 98us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:28.323694 (+ 85us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:28.324589 (+ 895us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:28.324724 (+ 135us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0504 13:56:28.324765 (+ 41us) write_op.cc:312] APPLY: finished
0504 13:56:28.324933 (+ 168us) log.cc:844] Serialized 29 byte log entry
0504 13:56:28.325065 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:28.325125 (+ 60us) write_op.cc:454] Released schema lock
0504 13:56:28.325153 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:28.325366 (+ 213us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:28.325389 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":557,"apply.run_cpu_time_us":1881,"apply.run_wall_time_us":1881,"num_ops":2,"prepare.queue_time_us":6375,"prepare.run_cpu_time_us":2167,"prepare.run_wall_time_us":2164,"raft.queue_time_us":5193,"raft.run_cpu_time_us":1197,"raft.run_wall_time_us":1196,"replication_time_us":16704}]]}
I20260504 13:56:28.361235 10431 raft_consensus.cc:493] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:28.361721 10431 raft_consensus.cc:515] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.364568 10431 leader_election.cc:290] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:28.364730 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fbe09b4078c4eebb52d7d00bc87371c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:28.368003 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9077fb9bb1544a2cbeb96f0d11c400db" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:28.368561 9681 raft_consensus.cc:2468] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:28.369333 10629 raft_consensus.cc:2749] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:28.371120 9473 leader_election.cc:304] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:28.373685 10705 raft_consensus.cc:2804] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:28.374017 10705 raft_consensus.cc:493] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:28.374295 10705 raft_consensus.cc:3060] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:28.381685 10705 raft_consensus.cc:515] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:28.393625 9768 leader_election.cc:343] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 030b35c3409143108c34ab1f0ee1b96d
I20260504 13:56:28.394528 9768 leader_election.cc:304] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:28.394616 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9077fb9bb1544a2cbeb96f0d11c400db" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:28.395700 10675 raft_consensus.cc:2749] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:56:28.404069 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 65927.2us
W20260504 13:56:28.405441 9474 leader_election.cc:343] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:28.410687 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9077fb9bb1544a2cbeb96f0d11c400db" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:28.414601 10431 raft_consensus.cc:493] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:28.415118 10431 raft_consensus.cc:515] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:28.419137 9473 leader_election.cc:343] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: a1aba9a2a58b4e6b9e87f8ec80fcb6f1
I20260504 13:56:28.419658 10431 leader_election.cc:290] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:28.425627 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet 604f1ba5f71041a7877daff8972a994a with delete_type TABLET_DATA_DELETED (replaced by tablet 2ddf890c17964e7f90409ebe3c4034ef) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:28.436904 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet c34630d55f4a4e19b33ddf4c13995a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:19 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:28.437525 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet afad9329cf7745ba8168b2a93f998295 with delete_type TABLET_DATA_DELETED (replaced by tablet af5c5a6cae4e4b6ea3eece2e21a587ec) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:28.438095 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet 50e129980cd84f6ca22cbf2addb06230 with delete_type TABLET_DATA_DELETED (replaced by tablet c47b6e3b4d0c45c28384f7aae6b749de) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:28.438686 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet 030b35c3409143108c34ab1f0ee1b96d with delete_type TABLET_DATA_DELETED (replaced by tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:28.439477 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet baf07a970a7c4a35b3c849b06d6c88d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:24 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:28.440236 9804 tablet_service.cc:1558] Processing DeleteTablet for tablet 930078a10afa4dd987ad021cf9609514 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:28.426926 9797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1a01231bb044433ae5827d07a6592f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.447321 10675 consensus_queue.cc:1048] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:28.447472 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_862ae682dfb44737a72e7c5df628a957 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:28.468505 9336 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57983fcecdb64a42884f797b4db57df0 has no consensus state
I20260504 13:56:28.468950 9805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a89bac060214d30924398007dfa1fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.469555 9336 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35768 (request call id 260) took 2766 ms. Trace:
I20260504 13:56:28.471836 9802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da61c91e10264fa49fe397d9a9b014bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.471767 10391 ts_tablet_manager.cc:1929] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:28.472139 10391 log.cc:1199] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/930078a10afa4dd987ad021cf9609514
I20260504 13:56:28.469929 9336 rpcz_store.cc:276] 0504 13:56:25.702623 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.157057 (+2454434us) service_pool.cc:224] Handling call
0504 13:56:28.229903 (+ 72846us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:28.230604 (+ 701us) catalog_manager.cc:3725] Apply alter schema
0504 13:56:28.231187 (+ 583us) catalog_manager.cc:3757] Acquired catalog manager lock
0504 13:56:28.245424 (+ 14237us) catalog_manager.cc:3915] Updating metadata on disk
0504 13:56:28.468085 (+222661us) catalog_manager.cc:3946] Committing alterations to in-memory state
0504 13:56:28.469476 (+ 1391us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:28.336010 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:28.336458 (+ 448us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:28.336486 (+ 28us) write_op.cc:435] Acquired schema lock
0504 13:56:28.336501 (+ 15us) tablet.cc:662] Decoding operations
0504 13:56:28.336779 (+ 278us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:28.336819 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:28.336844 (+ 25us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:28.337013 (+ 169us) tablet.cc:701] Row locks acquired
0504 13:56:28.337029 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:28.337137 (+ 108us) write_op.cc:270] Start()
0504 13:56:28.337218 (+ 81us) write_op.cc:276] Timestamp: P: 1777902988337040 usec, L: 0
0504 13:56:28.337237 (+ 19us) op_driver.cc:348] REPLICATION: starting
0504 13:56:28.383811 (+ 46574us) log.cc:844] Serialized 405 byte log entry
0504 13:56:28.385469 (+ 1658us) spinlock_profiling.cc:243] Waited 46 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b801951 00007fc27b894208 00007fc27bd069f1 00007fc27bd061c8 00007fc27bd0ae61 00007fc27bd0ae19 00007fc27bd0ada9 00007fc27bd0ad71 00007fc27bd09f9d 00000000005b3bc4 00000000005b39f9
0504 13:56:28.462803 (+ 77334us) op_driver.cc:464] REPLICATION: finished
0504 13:56:28.463289 (+ 486us) write_op.cc:301] APPLY: starting
0504 13:56:28.463362 (+ 73us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:28.463460 (+ 98us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:28.463485 (+ 25us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:28.463660 (+ 175us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:28.463814 (+ 154us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:28.463864 (+ 50us) write_op.cc:312] APPLY: finished
0504 13:56:28.464049 (+ 185us) log.cc:844] Serialized 23 byte log entry
0504 13:56:28.464204 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:28.464268 (+ 64us) write_op.cc:454] Released schema lock
0504 13:56:28.464299 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:28.464567 (+ 268us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:28.464605 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":331,"apply.run_cpu_time_us":1217,"apply.run_wall_time_us":1217,"mutex_wait_us":40351,"num_ops":1,"prepare.queue_time_us":71857,"prepare.run_cpu_time_us":3434,"prepare.run_wall_time_us":49648,"raft.queue_time_us":3473,"raft.run_cpu_time_us":756,"raft.run_wall_time_us":1457,"replication_time_us":125440,"spinlock_wait_cycles":105859328}]]}
I20260504 13:56:28.475137 9337 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35768 (request call id 262) took 2772 ms. Trace:
I20260504 13:56:28.476308 9337 rpcz_store.cc:276] 0504 13:56:25.703112 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.197228 (+2494116us) service_pool.cc:224] Handling call
0504 13:56:28.243924 (+ 46696us) catalog_manager.cc:2101] Acquired catalog manager lock
0504 13:56:28.255422 (+ 11498us) catalog_manager.cc:2147] Created new table and tablet info
0504 13:56:28.474372 (+218950us) catalog_manager.cc:2207] Wrote table and tablets to system table
0504 13:56:28.474762 (+ 390us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0504 13:56:28.475070 (+ 308us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:28.385791 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:28.386316 (+ 525us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:28.386347 (+ 31us) write_op.cc:435] Acquired schema lock
0504 13:56:28.386359 (+ 12us) tablet.cc:662] Decoding operations
0504 13:56:28.386649 (+ 290us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:28.386684 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:28.386734 (+ 50us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:28.387014 (+ 280us) tablet.cc:701] Row locks acquired
0504 13:56:28.387027 (+ 13us) write_op.cc:260] PREPARE: finished
0504 13:56:28.387133 (+ 106us) write_op.cc:270] Start()
0504 13:56:28.387395 (+ 262us) write_op.cc:276] Timestamp: P: 1777902988387034 usec, L: 0
0504 13:56:28.387423 (+ 28us) op_driver.cc:348] REPLICATION: starting
0504 13:56:28.388057 (+ 634us) log.cc:844] Serialized 433 byte log entry
0504 13:56:28.463052 (+ 74995us) op_driver.cc:464] REPLICATION: finished
0504 13:56:28.464914 (+ 1862us) write_op.cc:301] APPLY: starting
0504 13:56:28.464983 (+ 69us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:28.465119 (+ 136us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:28.465257 (+ 138us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:28.465858 (+ 601us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:28.466036 (+ 178us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0504 13:56:28.466105 (+ 69us) write_op.cc:312] APPLY: finished
0504 13:56:28.466281 (+ 176us) log.cc:844] Serialized 29 byte log entry
0504 13:56:28.466395 (+ 114us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:28.466448 (+ 53us) write_op.cc:454] Released schema lock
0504 13:56:28.466474 (+ 26us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1691,"apply.run_cpu_time_us":1714,"apply.run_wall_time_us":1715,"mutex_wait_us":100355,"num_ops":2,"prepare.queue_time_us":129223,"prepare.run_cpu_time_us":2341,"prepare.run_wall_time_us":43449,"raft.queue_time_us":3197,"raft.run_cpu_time_us":581,"raft.run_wall_time_us":1200,"replication_time_us":75544}]]}
I20260504 13:56:28.480141 9806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cb30835150949f39f6e7e7563381763. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:28.480499 9328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 123445us
I20260504 13:56:28.482760 9801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b84c3bedc7a84e0d8a873fb15689b314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.490031 10013 tablet_bootstrap.cc:492] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:28.490618 10013 ts_tablet_manager.cc:1403] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.265s user 0.005s sys 0.004s
I20260504 13:56:28.493046 9799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21d6358b451d4b8d87a22e30e0780f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.493957 9812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e53381ee0e7f4e9781b833b435bca831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.495684 9931 ts_tablet_manager.cc:1434] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.315s user 0.005s sys 0.002s
I20260504 13:56:28.496551 9931 tablet_bootstrap.cc:492] T 7eaa40eb4b1c425a9edfb7eaba953b65 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:28.498356 9338 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35768 (request call id 263) took 2780 ms. Trace:
I20260504 13:56:28.475941 9336 master_service.cc:954] ReplaceTablet: received request to replace tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:28.498714 9338 rpcz_store.cc:276] 0504 13:56:25.718374 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.288941 (+2570567us) service_pool.cc:224] Handling call
0504 13:56:28.291042 (+ 2101us) catalog_manager.cc:2101] Acquired catalog manager lock
0504 13:56:28.427903 (+136861us) catalog_manager.cc:2147] Created new table and tablet info
0504 13:56:28.497790 (+ 69887us) catalog_manager.cc:2207] Wrote table and tablets to system table
0504 13:56:28.498065 (+ 275us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0504 13:56:28.498289 (+ 224us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:28.429394 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:28.430065 (+ 671us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:28.430100 (+ 35us) write_op.cc:435] Acquired schema lock
0504 13:56:28.430122 (+ 22us) tablet.cc:662] Decoding operations
0504 13:56:28.430409 (+ 287us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:28.430448 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:28.430473 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:28.430688 (+ 215us) tablet.cc:701] Row locks acquired
0504 13:56:28.430712 (+ 24us) write_op.cc:260] PREPARE: finished
0504 13:56:28.430823 (+ 111us) write_op.cc:270] Start()
0504 13:56:28.430903 (+ 80us) write_op.cc:276] Timestamp: P: 1777902988430720 usec, L: 0
0504 13:56:28.430918 (+ 15us) op_driver.cc:348] REPLICATION: starting
0504 13:56:28.431414 (+ 496us) log.cc:844] Serialized 433 byte log entry
0504 13:56:28.489261 (+ 57847us) op_driver.cc:464] REPLICATION: finished
0504 13:56:28.495434 (+ 6173us) write_op.cc:301] APPLY: starting
0504 13:56:28.495487 (+ 53us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:28.495583 (+ 96us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:28.495672 (+ 89us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:28.496167 (+ 495us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:28.496305 (+ 138us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0504 13:56:28.496354 (+ 49us) write_op.cc:312] APPLY: finished
0504 13:56:28.496607 (+ 253us) log.cc:844] Serialized 29 byte log entry
0504 13:56:28.497207 (+ 600us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:28.497272 (+ 65us) write_op.cc:454] Released schema lock
0504 13:56:28.497300 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:28.497520 (+ 220us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:28.497544 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6007,"apply.run_cpu_time_us":1479,"apply.run_wall_time_us":2059,"num_ops":2,"prepare.queue_time_us":311,"prepare.run_cpu_time_us":2300,"prepare.run_wall_time_us":24863,"replication_time_us":58235}]]}
I20260504 13:56:28.402522 10729 raft_consensus.cc:515] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.472829 10391 ts_tablet_manager.cc:1950] T 930078a10afa4dd987ad021cf9609514 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:28.508517 10705 leader_election.cc:290] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:28.508806 9932 tablet_bootstrap.cc:492] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:28.480196 9342 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_276161949f364064b8702a62c6beb9e4" } modify_external_catalogs: true
I20260504 13:56:28.508749 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9077fb9bb1544a2cbeb96f0d11c400db" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:28.509250 9932 ts_tablet_manager.cc:1403] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.194s user 0.030s sys 0.022s
I20260504 13:56:28.509346 9342 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_276161949f364064b8702a62c6beb9e4" } modify_external_catalogs: true
I20260504 13:56:28.509438 9682 raft_consensus.cc:3060] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:28.511816 9932 raft_consensus.cc:359] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:28.512414 9932 raft_consensus.cc:385] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:28.514568 9932 raft_consensus.cc:740] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:28.515702 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 162) took 5256 ms. Trace:
I20260504 13:56:28.516019 10391 rpcz_store.cc:276] 0504 13:56:23.259535 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:23.287103 (+ 27568us) service_pool.cc:224] Handling call
0504 13:56:28.106983 (+4819880us) ts_tablet_manager.cc:1151] Deleting tablet 930078a10afa4dd987ad021cf9609514
0504 13:56:28.107023 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:28.249553 (+142530us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:28.471644 (+222091us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:28.515649 (+ 44005us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6328,"tablet-delete.queue_time_us":4819334}
I20260504 13:56:28.516543 9682 raft_consensus.cc:2468] T 9077fb9bb1544a2cbeb96f0d11c400db P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:28.515198 9932 consensus_queue.cc:260] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:28.517690 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 165) took 4823 ms. Trace:
I20260504 13:56:28.518148 9473 leader_election.cc:304] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:28.518051 10391 rpcz_store.cc:276] 0504 13:56:23.694041 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:23.694301 (+ 260us) service_pool.cc:224] Handling call
0504 13:56:28.517316 (+4823015us) ts_tablet_manager.cc:1151] Deleting tablet 8e4e583afc804212bece2f727411cc14
0504 13:56:28.517352 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:28.517619 (+ 267us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4815290}
I20260504 13:56:28.519248 9800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a862a43c5ac443d9bac98e0472ca23b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.521634 10703 raft_consensus.cc:2804] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:28.522048 10703 raft_consensus.cc:697] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:28.522794 10703 consensus_queue.cc:237] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.523480 9795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2bfb681e6824e9f9bdcbea23061d57a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.523634 10013 raft_consensus.cc:359] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:28.524327 10013 raft_consensus.cc:385] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:28.524600 10013 raft_consensus.cc:740] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:28.525898 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 930078a10afa4dd987ad021cf9609514 (table default.Table_0d2ce6c9ff35497b82dd58da3f913ecb [id=5a0c5354346746c186e36fe23406442c]) successfully deleted
W20260504 13:56:28.526551 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 8e4e583afc804212bece2f727411cc14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e4e583afc804212bece2f727411cc14 already in progress: creating tablet
I20260504 13:56:28.531445 9932 ts_tablet_manager.cc:1434] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.022s user 0.008s sys 0.000s
I20260504 13:56:28.534657 9932 tablet_bootstrap.cc:492] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:28.540349 10743 tablet_replica.cc:333] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:28.541356 10743 raft_consensus.cc:2243] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:28.541941 10743 raft_consensus.cc:2272] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:28.543979 9474 leader_election.cc:343] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:28.548066 10391 tablet_replica.cc:333] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:28.548887 10675 consensus_queue.cc:1048] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:28.549901 10391 raft_consensus.cc:2243] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:28.550043 9932 tablet_bootstrap.cc:654] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:28.548534 10013 consensus_queue.cc:260] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:28.552695 10683 raft_consensus.cc:493] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:28.553131 10683 raft_consensus.cc:515] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.554689 9340 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35768 (request call id 264) took 2653 ms. Trace:
I20260504 13:56:28.554864 10683 leader_election.cc:290] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:28.555823 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d63f665166466dba1e7e36c5e19506" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:28.557525 9620 leader_election.cc:343] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:28.557763 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d63f665166466dba1e7e36c5e19506" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:28.559015 9618 leader_election.cc:343] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:28.559410 9618 leader_election.cc:304] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:28.560082 10681 raft_consensus.cc:2749] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:28.555085 9340 rpcz_store.cc:276] 0504 13:56:25.901757 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.302544 (+2400787us) service_pool.cc:224] Handling call
0504 13:56:28.449145 (+146601us) catalog_manager.cc:2101] Acquired catalog manager lock
0504 13:56:28.452753 (+ 3608us) catalog_manager.cc:2147] Created new table and tablet info
0504 13:56:28.554114 (+101361us) catalog_manager.cc:2207] Wrote table and tablets to system table
0504 13:56:28.554378 (+ 264us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0504 13:56:28.554638 (+ 260us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:28.454410 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:28.454983 (+ 573us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:28.455039 (+ 56us) write_op.cc:435] Acquired schema lock
0504 13:56:28.455058 (+ 19us) tablet.cc:662] Decoding operations
0504 13:56:28.518133 (+ 63075us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:28.518176 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:28.518205 (+ 29us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:28.518407 (+ 202us) tablet.cc:701] Row locks acquired
0504 13:56:28.518423 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:28.518543 (+ 120us) write_op.cc:270] Start()
0504 13:56:28.518636 (+ 93us) write_op.cc:276] Timestamp: P: 1777902988518430 usec, L: 0
0504 13:56:28.518654 (+ 18us) op_driver.cc:348] REPLICATION: starting
0504 13:56:28.519206 (+ 552us) log.cc:844] Serialized 433 byte log entry
0504 13:56:28.543508 (+ 24302us) op_driver.cc:464] REPLICATION: finished
0504 13:56:28.551000 (+ 7492us) write_op.cc:301] APPLY: starting
0504 13:56:28.551079 (+ 79us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:28.551284 (+ 205us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:28.551368 (+ 84us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:28.551886 (+ 518us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:28.552013 (+ 127us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0504 13:56:28.552062 (+ 49us) write_op.cc:312] APPLY: finished
0504 13:56:28.552216 (+ 154us) log.cc:844] Serialized 29 byte log entry
0504 13:56:28.553710 (+ 1494us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:28.553772 (+ 62us) write_op.cc:454] Released schema lock
0504 13:56:28.553799 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:28.556434 (+ 2635us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:28.556460 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":7329,"apply.run_cpu_time_us":1448,"apply.run_wall_time_us":2983,"num_ops":2,"prepare.queue_time_us":462,"prepare.run_cpu_time_us":2572,"prepare.run_wall_time_us":102022,"raft.queue_time_us":361,"raft.run_cpu_time_us":668,"raft.run_wall_time_us":1786,"replication_time_us":24764}]]}
I20260504 13:56:28.572127 9932 tablet_bootstrap.cc:492] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:28.572516 9932 ts_tablet_manager.cc:1403] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.005s
I20260504 13:56:28.574708 9932 raft_consensus.cc:359] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.575311 9932 raft_consensus.cc:385] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:28.575544 9932 raft_consensus.cc:740] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:28.575224 10703 consensus_queue.cc:1048] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:28.576205 9932 consensus_queue.cc:260] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.580468 10391 raft_consensus.cc:2272] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:28.582432 9932 ts_tablet_manager.cc:1434] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260504 13:56:28.583155 10391 ts_tablet_manager.cc:1916] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:28.587414 9932 tablet_bootstrap.cc:492] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:28.595359 9932 tablet_bootstrap.cc:654] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:28.599943 10013 ts_tablet_manager.cc:1434] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.109s user 0.005s sys 0.003s
I20260504 13:56:28.607394 9932 tablet_bootstrap.cc:492] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:28.608239 9932 ts_tablet_manager.cc:1403] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.005s
I20260504 13:56:28.621181 10013 tablet_bootstrap.cc:492] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:28.626487 9932 raft_consensus.cc:359] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.627090 9932 raft_consensus.cc:385] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:28.627807 9932 raft_consensus.cc:740] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:28.628490 9932 consensus_queue.cc:260] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.630791 10705 consensus_queue.cc:1048] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:28.632582 9932 ts_tablet_manager.cc:1434] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260504 13:56:28.638774 9932 tablet_bootstrap.cc:492] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:28.642076 10743 ts_tablet_manager.cc:1916] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:28.644618 9932 tablet_bootstrap.cc:654] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:28.650698 9768 leader_election.cc:343] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: a1aba9a2a58b4e6b9e87f8ec80fcb6f1
I20260504 13:56:28.652882 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3404ba04d3ec4a44a5e625536b941b58" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:28.654403 9683 raft_consensus.cc:2468] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:28.655961 9932 tablet_bootstrap.cc:492] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:28.656404 9932 ts_tablet_manager.cc:1403] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.003s
I20260504 13:56:28.658679 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fbe09b4078c4eebb52d7d00bc87371c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:28.659439 9537 raft_consensus.cc:2393] T 1fbe09b4078c4eebb52d7d00bc87371c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 5beceb6317444247b31b44a6581707c6 in this term.
I20260504 13:56:28.661224 9932 raft_consensus.cc:359] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.661861 9932 raft_consensus.cc:385] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:28.662155 9932 raft_consensus.cc:740] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:28.662931 9932 consensus_queue.cc:260] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.666675 9932 ts_tablet_manager.cc:1434] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260504 13:56:28.669466 9932 tablet_bootstrap.cc:492] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:28.676954 9932 tablet_bootstrap.cc:654] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:28.683034 10683 consensus_queue.cc:1048] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:28.690685 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_6e41cb79b95848ba8c7b1f05f961c7f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:28.692132 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e41cb79b95848ba8c7b1f05f961c7f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:28.696568 9337 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_583272d3574a4ca78d056fe7d6b922bc" } new_table_name: "default.Table_d32487a6e29d4021b8ea3bab67b31164" modify_external_catalogs: true
I20260504 13:56:28.698534 10683 raft_consensus.cc:493] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:28.698935 10683 raft_consensus.cc:515] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.701732 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf1504f0202481ba8b08f9bec5bc3d9" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:28.702888 9618 leader_election.cc:343] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:28.703397 10683 leader_election.cc:290] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:28.705314 10013 tablet_bootstrap.cc:654] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:28.706632 10683 raft_consensus.cc:493] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:28.707043 10683 raft_consensus.cc:515] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.710199 10683 leader_election.cc:290] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:28.709760 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ebc96408d64c2da7dc6c4fc158995c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:28.716604 9618 leader_election.cc:343] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:28.713901 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3404ba04d3ec4a44a5e625536b941b58" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:28.718405 9808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92480654807d495b879576b0195871bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:28.721376 9831 raft_consensus.cc:3060] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:28.724248 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_10e92469b9cb4e9b965d0a10cfeecca8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:28.726045 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10e92469b9cb4e9b965d0a10cfeecca8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:28.747509 10703 consensus_queue.cc:1048] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:28.751579 9337 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5cb30835150949f39f6e7e7563381763 consensus state has no leader
I20260504 13:56:28.751786 10683 raft_consensus.cc:493] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:28.752379 10431 raft_consensus.cc:493] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:28.752208 10683 raft_consensus.cc:515] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.752768 10431 raft_consensus.cc:515] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.753021 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_2dc0381bd6c84ab59d5bb422de89c0e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:28.754047 10683 leader_election.cc:290] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
W20260504 13:56:28.754467 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2dc0381bd6c84ab59d5bb422de89c0e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:28.754730 10431 leader_election.cc:290] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:28.754987 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c12ffa802b840b1937f6f796083c478" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:28.756498 9680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a985338790e4887af38c49ba013417f" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:28.757048 9680 raft_consensus.cc:2468] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
W20260504 13:56:28.757210 9618 leader_election.cc:343] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:28.757767 10683 raft_consensus.cc:493] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:28.758104 9473 leader_election.cc:304] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:28.758251 10683 raft_consensus.cc:515] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.758764 10431 raft_consensus.cc:2804] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:28.759109 10431 raft_consensus.cc:493] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:28.759435 10431 raft_consensus.cc:3060] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:28.761161 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac3e36cfa4848a1ab015f89c56c2740" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:28.763947 9618 leader_election.cc:343] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:28.764712 10683 leader_election.cc:290] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:28.766340 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1aba9a2a58b4e6b9e87f8ec80fcb6f1" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:28.766908 9535 raft_consensus.cc:2468] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:28.768952 10431 raft_consensus.cc:515] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.771571 10431 leader_election.cc:290] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:28.771723 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a985338790e4887af38c49ba013417f" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:28.772329 9686 raft_consensus.cc:3060] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:28.712361 10729 leader_election.cc:290] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:28.711102 10728 leader_election.cc:290] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:28.780109 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf1504f0202481ba8b08f9bec5bc3d9" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:28.785552 9824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a985338790e4887af38c49ba013417f" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:28.786875 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a985338790e4887af38c49ba013417f" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:28.789808 9474 leader_election.cc:343] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:28.791134 9474 leader_election.cc:343] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:28.768944 9768 leader_election.cc:304] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:28.784009 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1aba9a2a58b4e6b9e87f8ec80fcb6f1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:28.807067 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ebc96408d64c2da7dc6c4fc158995c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:28.812930 9686 raft_consensus.cc:2468] T 8a985338790e4887af38c49ba013417f P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:28.820613 9473 leader_election.cc:304] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:28.821746 10431 raft_consensus.cc:2804] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:28.822190 10431 raft_consensus.cc:697] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:28.822970 10431 consensus_queue.cc:237] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.829649 10683 raft_consensus.cc:493] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:28.830200 10683 raft_consensus.cc:515] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:28.832525 10683 leader_election.cc:290] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:28.834046 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb77106b48a45a2aaa938dc474fa8bd" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:28.835368 9618 leader_election.cc:343] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:28.844415 10431 raft_consensus.cc:493] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:28.844928 10431 raft_consensus.cc:515] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:28.847823 10431 leader_election.cc:290] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:28.862809 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60d3cc59ffe947fbb57b7d3301ccbac1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:28.863533 9686 raft_consensus.cc:2468] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:28.865759 9473 leader_election.cc:304] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:28.866552 10431 raft_consensus.cc:2804] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:28.866914 10431 raft_consensus.cc:493] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260504 13:56:28.867265 9331 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5cb30835150949f39f6e7e7563381763 consensus state has no leader
I20260504 13:56:28.867280 10431 raft_consensus.cc:3060] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:28.874101 10431 raft_consensus.cc:515] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:28.876711 10431 leader_election.cc:290] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:28.882293 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60d3cc59ffe947fbb57b7d3301ccbac1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:28.882809 9686 raft_consensus.cc:3060] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:28.886533 10013 tablet_bootstrap.cc:492] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:28.887147 10013 ts_tablet_manager.cc:1403] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.266s user 0.002s sys 0.009s
I20260504 13:56:28.890372 10013 raft_consensus.cc:359] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.892756 9686 raft_consensus.cc:2468] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:28.913925 9932 tablet_bootstrap.cc:492] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:28.914343 9932 ts_tablet_manager.cc:1403] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.247s user 0.024s sys 0.070s
I20260504 13:56:28.916675 9932 raft_consensus.cc:359] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.917258 9932 raft_consensus.cc:385] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:28.917495 9932 raft_consensus.cc:740] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:28.918277 9932 consensus_queue.cc:260] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.920260 9932 ts_tablet_manager.cc:1434] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260504 13:56:28.923727 9932 tablet_bootstrap.cc:492] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:28.930739 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_2bed5ea8f43149e8bc3b21cd9d1b3846"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:28.931658 9932 tablet_bootstrap.cc:654] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:28.932775 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bed5ea8f43149e8bc3b21cd9d1b3846 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:28.893590 9473 leader_election.cc:304] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:28.940135 10013 raft_consensus.cc:385] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260504 13:56:28.940331 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 46808.5us
I20260504 13:56:28.940455 10013 raft_consensus.cc:740] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:28.941111 10431 raft_consensus.cc:2804] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:28.941520 10431 raft_consensus.cc:697] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:28.941190 10013 consensus_queue.cc:260] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.942183 10431 consensus_queue.cc:237] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:28.942790 9932 tablet_bootstrap.cc:492] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:28.943156 9932 ts_tablet_manager.cc:1403] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.007s
I20260504 13:56:28.944337 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_09fd612a3c4a4303b8f05530e7d49819"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:28.945550 9932 raft_consensus.cc:359] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.946017 10013 ts_tablet_manager.cc:1434] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.058s user 0.001s sys 0.004s
I20260504 13:56:28.946132 9932 raft_consensus.cc:385] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260504 13:56:28.946264 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09fd612a3c4a4303b8f05530e7d49819 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:28.946352 9932 raft_consensus.cc:740] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:28.946923 9932 consensus_queue.cc:260] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.953383 9932 ts_tablet_manager.cc:1434] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20260504 13:56:28.955821 10013 tablet_bootstrap.cc:492] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:28.965566 9932 tablet_bootstrap.cc:492] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:28.971566 9932 tablet_bootstrap.cc:654] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:28.990600 10013 tablet_bootstrap.cc:654] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:28.991622 9932 tablet_bootstrap.cc:492] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:28.992123 9932 ts_tablet_manager.cc:1403] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.000s
I20260504 13:56:28.934108 10391 ts_tablet_manager.cc:1929] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:28.994024 10391 log.cc:1199] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/9ed4f5ea230a413a8dcdbb661ce173b6
I20260504 13:56:28.994750 10391 ts_tablet_manager.cc:1950] T 9ed4f5ea230a413a8dcdbb661ce173b6 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:28.994439 9932 raft_consensus.cc:359] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:28.995039 9932 raft_consensus.cc:385] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:28.995409 9932 raft_consensus.cc:740] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:28.996105 9932 consensus_queue.cc:260] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.001024 9932 ts_tablet_manager.cc:1434] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260504 13:56:29.003861 10013 tablet_bootstrap.cc:492] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:29.004289 10013 ts_tablet_manager.cc:1403] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.010s
I20260504 13:56:29.006503 10013 raft_consensus.cc:359] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.007192 10013 raft_consensus.cc:385] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:29.007480 10013 raft_consensus.cc:740] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:29.008193 10013 consensus_queue.cc:260] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.010010 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 170) took 5089 ms. Trace:
I20260504 13:56:29.010434 10391 rpcz_store.cc:276] 0504 13:56:23.920368 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:23.920995 (+ 627us) service_pool.cc:224] Handling call
0504 13:56:28.547922 (+4626927us) ts_tablet_manager.cc:1151] Deleting tablet 9ed4f5ea230a413a8dcdbb661ce173b6
0504 13:56:28.547960 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:28.815299 (+267339us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:28.933985 (+118686us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.009927 (+ 75942us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4626202}
I20260504 13:56:29.011766 10391 tablet_replica.cc:333] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:29.011837 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 9ed4f5ea230a413a8dcdbb661ce173b6 (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]) successfully deleted
I20260504 13:56:29.012072 10013 ts_tablet_manager.cc:1434] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260504 13:56:29.012928 10013 tablet_bootstrap.cc:492] T 3404ba04d3ec4a44a5e625536b941b58 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:29.016144 9338 master_service.cc:954] ReplaceTablet: received request to replace tablet da61c91e10264fa49fe397d9a9b014bb from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:29.017148 9932 tablet_bootstrap.cc:492] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:29.012904 10391 raft_consensus.cc:2243] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:29.020186 10391 raft_consensus.cc:2272] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:29.021904 10391 ts_tablet_manager.cc:1916] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:29.024824 9517 tablet_service.cc:1511] Processing CreateTablet for tablet 75a43e11c8b34bd7bcddbf6e9b8f30dc (DEFAULT_TABLE table=default.Table_09255abca80f41dc895d81ccd89b9c7a [id=24485cf365ad4f9b89fa0f716fdb34cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.025729 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75a43e11c8b34bd7bcddbf6e9b8f30dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.028764 9657 tablet_service.cc:1511] Processing CreateTablet for tablet 75a43e11c8b34bd7bcddbf6e9b8f30dc (DEFAULT_TABLE table=default.Table_09255abca80f41dc895d81ccd89b9c7a [id=24485cf365ad4f9b89fa0f716fdb34cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.029733 9657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75a43e11c8b34bd7bcddbf6e9b8f30dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.030767 10013 tablet_bootstrap.cc:654] T 3404ba04d3ec4a44a5e625536b941b58 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:29.032084 9655 tablet_service.cc:1511] Processing CreateTablet for tablet 57983fcecdb64a42884f797b4db57df0 (DEFAULT_TABLE table=default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.032213 9509 tablet_service.cc:1511] Processing CreateTablet for tablet 57983fcecdb64a42884f797b4db57df0 (DEFAULT_TABLE table=default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.033118 9655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57983fcecdb64a42884f797b4db57df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.033197 9509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57983fcecdb64a42884f797b4db57df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.052573 10391 ts_tablet_manager.cc:1929] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:29.052922 10391 log.cc:1199] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/1501455a0af348e38746d100987afdd0
I20260504 13:56:29.053578 10391 ts_tablet_manager.cc:1950] T 1501455a0af348e38746d100987afdd0 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:29.055092 9344 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 1fbe09b4078c4eebb52d7d00bc87371c (table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee]). Expected version 1 got 0
I20260504 13:56:29.055351 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 172) took 4899 ms. Trace:
I20260504 13:56:29.055639 10391 rpcz_store.cc:276] 0504 13:56:24.155870 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:24.178538 (+ 22668us) service_pool.cc:224] Handling call
0504 13:56:29.011514 (+4832976us) ts_tablet_manager.cc:1151] Deleting tablet 1501455a0af348e38746d100987afdd0
0504 13:56:29.011549 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:29.035891 (+ 24342us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.052463 (+ 16572us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.055296 (+ 2833us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4719277}
I20260504 13:56:29.056782 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 1501455a0af348e38746d100987afdd0 (table default.Table_9b8d10cfdddc499a86ba289a87dbe689 [id=1fcd0497578846cb9bbd71e501606912]) successfully deleted
I20260504 13:56:29.057907 9344 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 274bb204041c4e1085395c7bd938aaf1 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]). Expected version 4 got 0
I20260504 13:56:29.058447 10013 tablet_bootstrap.cc:492] T 3404ba04d3ec4a44a5e625536b941b58 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:29.058826 10013 ts_tablet_manager.cc:1403] T 3404ba04d3ec4a44a5e625536b941b58 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.000s
I20260504 13:56:29.061023 10013 raft_consensus.cc:359] T 3404ba04d3ec4a44a5e625536b941b58 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.061579 10013 raft_consensus.cc:385] T 3404ba04d3ec4a44a5e625536b941b58 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:29.062083 10013 raft_consensus.cc:740] T 3404ba04d3ec4a44a5e625536b941b58 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:29.062651 10013 consensus_queue.cc:260] T 3404ba04d3ec4a44a5e625536b941b58 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.065516 9932 tablet_bootstrap.cc:654] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:29.066506 10013 ts_tablet_manager.cc:1434] T 3404ba04d3ec4a44a5e625536b941b58 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260504 13:56:29.068063 9344 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:29.068959 10013 tablet_bootstrap.cc:492] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:29.076112 10391 tablet_replica.cc:333] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:29.077544 10391 raft_consensus.cc:2243] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:29.078356 10391 raft_consensus.cc:2272] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:29.080078 10391 ts_tablet_manager.cc:1916] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:29.089072 10013 tablet_bootstrap.cc:654] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:29.096167 9344 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 7eaa40eb4b1c425a9edfb7eaba953b65 (table default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e]). Expected version 1 got 0
I20260504 13:56:29.099328 9809 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 204) took 1320 ms. Trace:
I20260504 13:56:29.106633 9809 rpcz_store.cc:276] 0504 13:56:27.778764 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.214167 (+435403us) service_pool.cc:224] Handling call
0504 13:56:28.306720 (+ 92553us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.306789 (+ 69us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:29.088622 (+781833us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.095208 (+ 6586us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.099269 (+ 4061us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:29.110215 9344 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 8db7c4732fa048f9950248f45d144c57 (table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62]). Expected version 1 got 0
I20260504 13:56:29.114135 10013 tablet_bootstrap.cc:492] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:29.114511 10013 ts_tablet_manager.cc:1403] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.007s
I20260504 13:56:29.117174 10013 raft_consensus.cc:359] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:29.117786 10013 raft_consensus.cc:385] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:29.118046 10013 raft_consensus.cc:740] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:29.122169 10391 ts_tablet_manager.cc:1929] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:29.122493 10391 log.cc:1199] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/baf07a970a7c4a35b3c849b06d6c88d0
I20260504 13:56:29.123210 10391 ts_tablet_manager.cc:1950] T baf07a970a7c4a35b3c849b06d6c88d0 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:29.124935 10013 consensus_queue.cc:260] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:29.129696 10013 ts_tablet_manager.cc:1434] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.015s user 0.005s sys 0.002s
I20260504 13:56:29.134567 10013 tablet_bootstrap.cc:492] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:28.908252 9813 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 202) took 1228 ms. Trace:
I20260504 13:56:29.135895 9813 rpcz_store.cc:276] 0504 13:56:27.679995 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.204727 (+524732us) service_pool.cc:224] Handling call
0504 13:56:28.305845 (+101118us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.305924 (+ 79us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:28.702449 (+396525us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:28.861879 (+159430us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:28.908197 (+ 46318us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:29.152846 9337 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_0eae04160cf241df95cc53da6f4dfacf" } new_table_name: "default.Table_d58041edcf8545f592c4a66547dd4a75" modify_external_catalogs: true
I20260504 13:56:29.154736 10013 tablet_bootstrap.cc:654] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:29.156809 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 174) took 4933 ms. Trace:
W20260504 13:56:28.842142 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 109409us
I20260504 13:56:28.843075 9830 raft_consensus.cc:2468] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:29.157546 10391 rpcz_store.cc:276] 0504 13:56:24.223196 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:24.223305 (+ 109us) service_pool.cc:224] Handling call
0504 13:56:29.075961 (+4852656us) ts_tablet_manager.cc:1151] Deleting tablet baf07a970a7c4a35b3c849b06d6c88d0
0504 13:56:29.076002 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:29.105008 (+ 29006us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.122041 (+ 17033us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.156752 (+ 34711us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":192,"tablet-delete.queue_time_us":4770445}
I20260504 13:56:28.909379 10629 raft_consensus.cc:2804] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:29.158905 10391 tablet_replica.cc:333] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:29.159677 10391 raft_consensus.cc:2243] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
W20260504 13:56:29.159585 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 251088us
I20260504 13:56:29.160480 10391 raft_consensus.cc:2272] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:29.161717 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet baf07a970a7c4a35b3c849b06d6c88d0 (table default.Table_2abb0f27da024032b5646af14e0946e8 [id=55f725fdab804a13b226f319cdc2fa34]) successfully deleted
I20260504 13:56:29.162220 10391 ts_tablet_manager.cc:1916] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:29.163610 9507 tablet_service.cc:1511] Processing CreateTablet for tablet 99fe335b131b45a587ec1cf526dd7f79 (DEFAULT_TABLE table=default.Table_655e2e8cf3544fbeb0ce33829bb2c8a0 [id=9db1e031520d405ea4989e30d5194911]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.163955 9344 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]). Expected version 2 got 0
I20260504 13:56:29.165194 9507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99fe335b131b45a587ec1cf526dd7f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.168601 9655 tablet_service.cc:1511] Processing CreateTablet for tablet 99fe335b131b45a587ec1cf526dd7f79 (DEFAULT_TABLE table=default.Table_655e2e8cf3544fbeb0ce33829bb2c8a0 [id=9db1e031520d405ea4989e30d5194911]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.169651 9655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99fe335b131b45a587ec1cf526dd7f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.170785 9512 tablet_service.cc:1511] Processing CreateTablet for tablet 03e1d998c4314db3b3682a0510393749 (DEFAULT_TABLE table=default.Table_f00a5967b1b7422b830dcf11db2acbd8 [id=6a34d9af17c84984b479bbd10025fba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.171751 9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03e1d998c4314db3b3682a0510393749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.172153 9807 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 205) took 1393 ms. Trace:
I20260504 13:56:29.172254 9657 tablet_service.cc:1511] Processing CreateTablet for tablet 03e1d998c4314db3b3682a0510393749 (DEFAULT_TABLE table=default.Table_f00a5967b1b7422b830dcf11db2acbd8 [id=6a34d9af17c84984b479bbd10025fba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.172459 9807 rpcz_store.cc:276] 0504 13:56:27.779094 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.215062 (+435968us) service_pool.cc:224] Handling call
0504 13:56:28.307112 (+ 92050us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.307197 (+ 85us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:28.926728 (+619531us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.104143 (+177415us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.172103 (+ 67960us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:29.173102 9657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03e1d998c4314db3b3682a0510393749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.159322 9824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60d3cc59ffe947fbb57b7d3301ccbac1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:29.160703 10629 raft_consensus.cc:493] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:29.191607 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac3e36cfa4848a1ab015f89c56c2740" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:29.195482 9336 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:29.196709 9336 catalog_manager.cc:6741] ReplaceTablet: tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 deleted and replaced by tablet eef975650610423c88ce8e4cfb0b013b
I20260504 13:56:29.197013 9336 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:35768 (request call id 266) took 1467 ms. Trace:
I20260504 13:56:29.197259 9336 rpcz_store.cc:276] 0504 13:56:27.729813 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.475798 (+745985us) service_pool.cc:224] Handling call
0504 13:56:29.196966 (+721168us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:29.175964 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:29.176395 (+ 431us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:29.176422 (+ 27us) write_op.cc:435] Acquired schema lock
0504 13:56:29.176438 (+ 16us) tablet.cc:662] Decoding operations
0504 13:56:29.177022 (+ 584us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:29.177065 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:29.177089 (+ 24us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:29.177318 (+ 229us) tablet.cc:701] Row locks acquired
0504 13:56:29.177335 (+ 17us) write_op.cc:260] PREPARE: finished
0504 13:56:29.177455 (+ 120us) write_op.cc:270] Start()
0504 13:56:29.177547 (+ 92us) write_op.cc:276] Timestamp: P: 1777902989177403 usec, L: 0
0504 13:56:29.177566 (+ 19us) op_driver.cc:348] REPLICATION: starting
0504 13:56:29.178095 (+ 529us) log.cc:844] Serialized 628 byte log entry
0504 13:56:29.186796 (+ 8701us) op_driver.cc:464] REPLICATION: finished
0504 13:56:29.193229 (+ 6433us) write_op.cc:301] APPLY: starting
0504 13:56:29.193283 (+ 54us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:29.193374 (+ 91us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:29.193465 (+ 91us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:29.193860 (+ 395us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:29.194056 (+ 196us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:29.194097 (+ 41us) write_op.cc:312] APPLY: finished
0504 13:56:29.194254 (+ 157us) log.cc:844] Serialized 29 byte log entry
0504 13:56:29.194820 (+ 566us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:29.194885 (+ 65us) write_op.cc:454] Released schema lock
0504 13:56:29.194914 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6289,"num_ops":2,"prepare.queue_time_us":171925,"raft.queue_time_us":78,"raft.run_cpu_time_us":545,"raft.run_wall_time_us":13372,"replication_time_us":9121}]]}
I20260504 13:56:29.201812 9516 tablet_service.cc:1558] Processing DeleteTablet for tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 with delete_type TABLET_DATA_DELETED (replaced by tablet eef975650610423c88ce8e4cfb0b013b) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:29.202423 9659 tablet_service.cc:1558] Processing DeleteTablet for tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 with delete_type TABLET_DATA_DELETED (replaced by tablet eef975650610423c88ce8e4cfb0b013b) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:29.161406 9767 leader_election.cc:304] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:29.208811 10013 tablet_bootstrap.cc:492] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:29.209205 10013 ts_tablet_manager.cc:1403] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.075s user 0.008s sys 0.001s
I20260504 13:56:29.165489 9344 catalog_manager.cc:5671] T 975fa1a4975242a388c15fe73abea1c7 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:29.219583 10013 raft_consensus.cc:359] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:29.220201 10013 raft_consensus.cc:385] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:29.220448 10013 raft_consensus.cc:740] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:29.186168 10629 raft_consensus.cc:3060] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:29.223510 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 62184.6us
I20260504 13:56:29.225880 9336 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_4842f50414d34bdb8a10802bc059b421"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:29.227576 9336 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4842f50414d34bdb8a10802bc059b421 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:29.236047 9517 tablet_service.cc:1511] Processing CreateTablet for tablet df71dc4f2cf84cda8cfc534a0eea86f8 (DEFAULT_TABLE table=default.Table_1611aa9f4dba40e09dae4433e2d20154 [id=2fb2c56e2fb44b9aac476e06e36e5778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.236753 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c12ffa802b840b1937f6f796083c478" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:29.220990 10013 consensus_queue.cc:260] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:29.187043 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60d3cc59ffe947fbb57b7d3301ccbac1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:29.238989 10788 raft_consensus.cc:2749] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260504 13:56:29.242293 10013 ts_tablet_manager.cc:1434] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.033s user 0.005s sys 0.001s
I20260504 13:56:29.249431 9810 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 207) took 1232 ms. Trace:
I20260504 13:56:29.249837 9810 rpcz_store.cc:276] 0504 13:56:28.017034 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.220558 (+203524us) service_pool.cc:224] Handling call
0504 13:56:28.321030 (+100472us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.321091 (+ 61us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:28.918308 (+597217us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.205877 (+287569us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.249377 (+ 43500us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:29.253499 9814 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 203) took 1475 ms. Trace:
I20260504 13:56:29.253825 9814 rpcz_store.cc:276] 0504 13:56:27.778396 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.207356 (+428960us) service_pool.cc:224] Handling call
0504 13:56:28.306314 (+ 98958us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.306380 (+ 66us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:29.085751 (+779371us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.253129 (+167378us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.253450 (+ 321us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:29.258119 9932 tablet_bootstrap.cc:492] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
W20260504 13:56:29.261854 9620 leader_election.cc:343] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.262177 9665 tablet_service.cc:1511] Processing CreateTablet for tablet 6dd139c347f7406aac340cc6ac994273 (DEFAULT_TABLE table=default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.263641 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb77106b48a45a2aaa938dc474fa8bd" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:29.263790 9620 leader_election.cc:304] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:29.264349 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd139c347f7406aac340cc6ac994273. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:29.264659 9620 leader_election.cc:343] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.264986 9620 leader_election.cc:304] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:29.265853 9620 leader_election.cc:343] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.266170 9620 leader_election.cc:304] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:29.266777 10767 raft_consensus.cc:2749] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:29.267313 10767 raft_consensus.cc:2749] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:29.267767 10767 raft_consensus.cc:2749] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:29.268703 9620 leader_election.cc:343] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.269035 9620 leader_election.cc:304] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:29.269590 10681 raft_consensus.cc:2749] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:29.261564 9657 tablet_service.cc:1511] Processing CreateTablet for tablet df71dc4f2cf84cda8cfc534a0eea86f8 (DEFAULT_TABLE table=default.Table_1611aa9f4dba40e09dae4433e2d20154 [id=2fb2c56e2fb44b9aac476e06e36e5778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.276396 9657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df71dc4f2cf84cda8cfc534a0eea86f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.276170 9474 leader_election.cc:304] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:29.278774 10703 raft_consensus.cc:2804] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:29.280622 10703 raft_consensus.cc:493] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:29.281064 10703 raft_consensus.cc:3060] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:29.284516 9338 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet da61c91e10264fa49fe397d9a9b014bb
I20260504 13:56:29.285857 9338 catalog_manager.cc:6741] ReplaceTablet: tablet da61c91e10264fa49fe397d9a9b014bb deleted and replaced by tablet 6323e1c10e7643f8a28f04e285f620e9
W20260504 13:56:29.287315 10771 log.cc:927] Time spent T 9f3905bfb30e4b03a7fcf0a2f57e2117 P d4051d2bf0bf410eb8acd107560df888: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
W20260504 13:56:29.289901 9474 leader_election.cc:343] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:29.291994 9474 leader_election.cc:343] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.296923 9655 tablet_service.cc:1558] Processing DeleteTablet for tablet da61c91e10264fa49fe397d9a9b014bb with delete_type TABLET_DATA_DELETED (replaced by tablet 6323e1c10e7643f8a28f04e285f620e9) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:29.298806 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_93e03451ea09440d847c57c026230518"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:29.300462 10790 tablet_replica.cc:333] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
W20260504 13:56:29.300410 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93e03451ea09440d847c57c026230518 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:29.309099 9932 ts_tablet_manager.cc:1403] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.292s user 0.023s sys 0.003s
I20260504 13:56:29.312322 10703 raft_consensus.cc:515] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.313835 10391 ts_tablet_manager.cc:1929] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:29.321878 9931 tablet_bootstrap.cc:654] T 7eaa40eb4b1c425a9edfb7eaba953b65 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:29.317570 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df71dc4f2cf84cda8cfc534a0eea86f8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:29.326467 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 102375us
W20260504 13:56:29.327111 9767 leader_election.cc:343] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.321211 10013 tablet_bootstrap.cc:492] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:29.328179 9767 leader_election.cc:304] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:29.338512 10013 tablet_bootstrap.cc:654] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:29.339828 9337 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a862a43c5ac443d9bac98e0472ca23b2 consensus state has no leader
I20260504 13:56:29.338994 10391 log.cc:1199] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/604f1ba5f71041a7877daff8972a994a
W20260504 13:56:29.344090 10747 log.cc:927] Time spent T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888: Append to log took a long time: real 0.245s user 0.001s sys 0.001s
I20260504 13:56:29.345037 10391 ts_tablet_manager.cc:1950] T 604f1ba5f71041a7877daff8972a994a P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:29.370620 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 175) took 5129 ms. Trace:
I20260504 13:56:29.370987 10391 rpcz_store.cc:276] 0504 13:56:24.241009 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:24.241118 (+ 109us) service_pool.cc:224] Handling call
0504 13:56:29.158719 (+4917601us) ts_tablet_manager.cc:1151] Deleting tablet 604f1ba5f71041a7877daff8972a994a
0504 13:56:29.158760 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:29.244830 (+ 86070us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.313723 (+ 68893us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.370557 (+ 56834us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2335,"tablet-delete.queue_time_us":4853036}
I20260504 13:56:29.371472 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1aba9a2a58b4e6b9e87f8ec80fcb6f1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:29.372052 9832 raft_consensus.cc:3060] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:29.372553 10391 tablet_replica.cc:333] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:29.373391 10391 raft_consensus.cc:2243] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:29.373780 10391 raft_consensus.cc:2272] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:29.377249 9509 tablet_service.cc:1511] Processing CreateTablet for tablet 6dd139c347f7406aac340cc6ac994273 (DEFAULT_TABLE table=default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.378417 9509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd139c347f7406aac340cc6ac994273. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.384843 10703 leader_election.cc:290] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:29.386323 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 58218.3us
I20260504 13:56:29.394353 10681 consensus_queue.cc:1048] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:29.396001 9516 tablet_service.cc:1558] Processing DeleteTablet for tablet da61c91e10264fa49fe397d9a9b014bb with delete_type TABLET_DATA_DELETED (replaced by tablet 6323e1c10e7643f8a28f04e285f620e9) from {username='slave'} at 127.0.0.1:52642
W20260504 13:56:29.396009 9620 leader_election.cc:343] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.396341 9620 leader_election.cc:304] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:29.397460 10681 raft_consensus.cc:2749] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:29.406087 9473 leader_election.cc:343] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: a1aba9a2a58b4e6b9e87f8ec80fcb6f1
I20260504 13:56:29.407107 10611 consensus_queue.cc:1048] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:29.419574 10790 raft_consensus.cc:2243] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:29.419775 10013 tablet_bootstrap.cc:492] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:29.420274 10013 ts_tablet_manager.cc:1403] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.099s user 0.005s sys 0.007s
I20260504 13:56:29.422513 9832 raft_consensus.cc:2468] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:29.422544 10013 raft_consensus.cc:359] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.423105 10013 raft_consensus.cc:385] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:29.423352 10013 raft_consensus.cc:740] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
W20260504 13:56:29.423712 9831 consensus_meta.cc:287] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888: Time spent flushing consensus metadata: real 0.702s user 0.007s sys 0.007s
I20260504 13:56:29.423859 10013 consensus_queue.cc:260] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.430943 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 604f1ba5f71041a7877daff8972a994a (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]) successfully deleted
I20260504 13:56:29.431829 9798 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 196) took 2108 ms. Trace:
I20260504 13:56:29.432174 9798 rpcz_store.cc:276] 0504 13:56:27.323861 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.336513 (+ 12652us) service_pool.cc:224] Handling call
0504 13:56:27.922747 (+586234us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b5800000098. stack: 00007fa12409cc83 00007fa12409cb80 00007fa123c0d574 000000000050e094 000000000050e029 00007fa123f0c941 00007fa1240916c1 00007fa124091572 00007fa12cede1d9 00007fa12c1475e0 00007fa12c22cef1 00007fa12c20a183 00007fa12c1b8ac0 00007fa12a777bf4 00007fa12a777b82 00007fa12a777ab1
0504 13:56:28.262885 (+340138us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.268411 (+ 5526us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:29.238841 (+970430us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.431324 (+192483us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.431758 (+ 434us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26779904}
I20260504 13:56:29.437439 9802 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 206) took 1593 ms. Trace:
I20260504 13:56:29.437850 9802 rpcz_store.cc:276] 0504 13:56:27.844367 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.218363 (+373996us) service_pool.cc:224] Handling call
0504 13:56:28.319283 (+100920us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.323795 (+ 4512us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:28.907969 (+584174us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.436967 (+528998us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.437375 (+ 408us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:29.440172 9474 leader_election.cc:304] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:29.445470 9812 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 191) took 2123 ms. Trace:
W20260504 13:56:29.445927 9474 raft_consensus.cc:2686] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260504 13:56:29.447145 10703 consensus_queue.cc:1048] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:29.450475 10391 ts_tablet_manager.cc:1916] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:29.317152 10629 raft_consensus.cc:515] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.452795 10629 leader_election.cc:290] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:29.450201 10013 ts_tablet_manager.cc:1434] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.030s user 0.004s sys 0.000s
I20260504 13:56:29.455679 10013 tablet_bootstrap.cc:492] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:29.455303 10734 raft_consensus.cc:493] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:29.456159 10734 raft_consensus.cc:515] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.445890 9812 rpcz_store.cc:276] 0504 13:56:27.322333 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.354966 (+ 32633us) service_pool.cc:224] Handling call
0504 13:56:27.921432 (+566466us) spinlock_profiling.cc:243] Waited 2.82 ms on lock 0x7b5800000098. stack: 00007fa12409cc83 00007fa12409cb80 00007fa123c0d574 000000000050e094 000000000050e029 00007fa123f0c941 00007fa1240916c1 00007fa124091572 00007fa12cede1d9 00007fa12c1475e0 00007fa12c22cef1 00007fa12c20a183 00007fa12c1b8ac0 00007fa12a777bf4 00007fa12a777b82 00007fa12a777ab1
0504 13:56:28.222193 (+300761us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.222286 (+ 93us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:29.428081 (+1205795us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.442676 (+ 14595us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.445394 (+ 2718us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6481152}
I20260504 13:56:29.467990 9808 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 188) took 2137 ms. Trace:
I20260504 13:56:29.468981 10790 raft_consensus.cc:2272] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:29.468305 9808 rpcz_store.cc:276] 0504 13:56:27.321468 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.352006 (+ 30538us) service_pool.cc:224] Handling call
0504 13:56:27.926758 (+574752us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b5800000098. stack: 00007fa12409cc83 00007fa12409cb80 00007fa123c0d574 000000000050e094 000000000050e029 00007fa123f0c941 00007fa1240916c1 00007fa124091572 00007fa12cede1d9 00007fa12c1475e0 00007fa12c22cef1 00007fa12c20a183 00007fa12c1b8ac0 00007fa12a777bf4 00007fa12a777b82 00007fa12a777ab1
0504 13:56:28.222843 (+296085us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.222928 (+ 85us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:29.207652 (+984724us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.444502 (+236850us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.458932 (+ 14430us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2760576}
I20260504 13:56:29.439098 10729 raft_consensus.cc:2764] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260504 13:56:29.444321 10675 consensus_queue.cc:1048] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.037s
I20260504 13:56:29.491549 9805 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 182) took 2172 ms. Trace:
I20260504 13:56:29.491948 9805 rpcz_store.cc:276] 0504 13:56:27.319610 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.347265 (+ 27655us) service_pool.cc:224] Handling call
0504 13:56:28.028853 (+681588us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b5800000098. stack: 00007fa12409cc83 00007fa12409cb80 00007fa123c0d574 000000000050e094 000000000050e029 00007fa123f0c941 00007fa1240916c1 00007fa124091572 00007fa12cede1d9 00007fa12c1475e0 00007fa12c22cef1 00007fa12c20a183 00007fa12c1b8ac0 00007fa12a777bf4 00007fa12a777b82 00007fa12a777ab1
0504 13:56:28.030343 (+ 1490us) spinlock_profiling.cc:243] Waited 937 us on lock 0x7b5800000098. stack: 00007fa12409cc83 00007fa12409cb80 00007fa123c0d574 000000000050e094 000000000050e029 00007fa123f0c941 00007fa1240919ec 00007fa1240915ab 00007fa12cede1d9 00007fa12c1475e0 00007fa12c22cef1 00007fa12c20a183 00007fa12c1b8ac0 00007fa12a777bf4 00007fa12a777b82 00007fa12a777ab1
0504 13:56:28.234508 (+204165us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.234624 (+ 116us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:29.337892 (+1103268us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.484273 (+146381us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.491484 (+ 7211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33487488}
I20260504 13:56:29.493624 9796 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 200) took 1822 ms. Trace:
I20260504 13:56:29.494000 9796 rpcz_store.cc:276] 0504 13:56:27.671439 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.682528 (+ 11089us) service_pool.cc:224] Handling call
0504 13:56:28.205668 (+523140us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.205771 (+ 103us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:29.333917 (+1128146us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.493141 (+159224us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.493570 (+ 429us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:29.510892 9803 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 189) took 2189 ms. Trace:
I20260504 13:56:29.511258 9803 rpcz_store.cc:276] 0504 13:56:27.321743 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.343861 (+ 22118us) service_pool.cc:224] Handling call
0504 13:56:27.988593 (+644732us) spinlock_profiling.cc:243] Waited 1.8 ms on lock 0x7b5800000098. stack: 00007fa12409cc83 00007fa12409cb80 00007fa123c0d574 000000000050e094 000000000050e029 00007fa123f0c941 00007fa1240916c1 00007fa124091572 00007fa12cede1d9 00007fa12c1475e0 00007fa12c22cef1 00007fa12c20a183 00007fa12c1b8ac0 00007fa12a777bf4 00007fa12a777b82 00007fa12a777ab1
0504 13:56:28.231345 (+242752us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.231707 (+ 362us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:29.268779 (+1037072us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.510440 (+241661us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.510820 (+ 380us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4140800}
I20260504 13:56:29.516019 10734 leader_election.cc:290] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:29.518337 10743 ts_tablet_manager.cc:1929] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:29.518791 10743 log.cc:1199] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/bd8cd1caaeb249e3bafee7f1f0d4df2f
I20260504 13:56:29.522001 10743 ts_tablet_manager.cc:1950] T bd8cd1caaeb249e3bafee7f1f0d4df2f P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:29.475930 9339 catalog_manager.cc:5671] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:29.528855 9339 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 1fbe09b4078c4eebb52d7d00bc87371c (table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee]). Expected version 1 got 0
I20260504 13:56:29.539597 9339 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 274bb204041c4e1085395c7bd938aaf1 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]). Expected version 4 got 0
I20260504 13:56:29.522586 9799 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 212) took 1462 ms. Trace:
I20260504 13:56:29.542649 9799 rpcz_store.cc:276] 0504 13:56:28.060195 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.336339 (+276144us) service_pool.cc:224] Handling call
0504 13:56:28.347381 (+ 11042us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.358306 (+ 10925us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:29.433139 (+1074833us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.522147 (+ 89008us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.522524 (+ 377us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:29.529045 9795 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 201) took 1849 ms. Trace:
I20260504 13:56:29.543586 9795 rpcz_store.cc:276] 0504 13:56:27.679631 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.685242 (+ 5611us) service_pool.cc:224] Handling call
0504 13:56:27.921986 (+236744us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b5800000098. stack: 00007fa12409cc83 00007fa12409cb80 00007fa123c0d574 000000000050e094 000000000050e029 00007fa123f0c941 00007fa1240916c1 00007fa124091572 00007fa12cede1d9 00007fa12c1475e0 00007fa12c22cef1 00007fa12c20a183 00007fa12c1b8ac0 00007fa12a777bf4 00007fa12a777b82 00007fa12a777ab1
0504 13:56:28.222465 (+300479us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.222567 (+ 102us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:29.451500 (+1228933us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.528594 (+ 77094us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.528990 (+ 396us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5288192}
I20260504 13:56:29.540845 9801 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 193) took 2218 ms. Trace:
I20260504 13:56:29.544690 9801 rpcz_store.cc:276] 0504 13:56:27.322900 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.340708 (+ 17808us) service_pool.cc:224] Handling call
0504 13:56:27.988345 (+647637us) spinlock_profiling.cc:243] Waited 5.62 ms on lock 0x7b5800000098. stack: 00007fa12409cc83 00007fa12409cb80 00007fa123c0d574 000000000050e094 000000000050e029 00007fa123f0c941 00007fa1240916c1 00007fa124091572 00007fa12cede1d9 00007fa12c1475e0 00007fa12c22cef1 00007fa12c20a183 00007fa12c1b8ac0 00007fa12a777bf4 00007fa12a777b82 00007fa12a777ab1
0504 13:56:28.224930 (+236585us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.225026 (+ 96us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:29.369885 (+1144859us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.519299 (+149414us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.540790 (+ 21491us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12925184}
I20260504 13:56:29.545382 10743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 208) took 1515 ms. Trace:
I20260504 13:56:29.471805 10790 ts_tablet_manager.cc:1916] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:29.547307 9806 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 180) took 2229 ms. Trace:
I20260504 13:56:29.548341 9800 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 194) took 2225 ms. Trace:
W20260504 13:56:29.557824 9474 consensus_peers.cc:597] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:29.558071 9800 rpcz_store.cc:276] 0504 13:56:27.323223 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.339287 (+ 16064us) service_pool.cc:224] Handling call
0504 13:56:27.789639 (+450352us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b5800000098. stack: 00007fa12409cc83 00007fa12409cb80 00007fa123c0d574 000000000050e094 000000000050e029 00007fa123f0c941 00007fa1240916c1 00007fa124091572 00007fa12cede1d9 00007fa12c1475e0 00007fa12c22cef1 00007fa12c20a183 00007fa12c1b8ac0 00007fa12a777bf4 00007fa12a777b82 00007fa12a777ab1
0504 13:56:28.208817 (+419178us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.212445 (+ 3628us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:29.425317 (+1212872us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.547876 (+122559us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.548290 (+ 414us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6554,"spinlock_wait_cycles":8441600}
I20260504 13:56:29.551899 9797 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:56380 (request call id 197) took 2227 ms. Trace:
I20260504 13:56:29.559345 9797 rpcz_store.cc:276] 0504 13:56:27.324175 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.334469 (+ 10294us) service_pool.cc:224] Handling call
0504 13:56:27.936117 (+601648us) spinlock_profiling.cc:243] Waited 4.28 ms on lock 0x7b5800000098. stack: 00007fa12409cc83 00007fa12409cb80 00007fa123c0d574 000000000050e094 000000000050e029 00007fa123f0c941 00007fa1240916c1 00007fa124091572 00007fa12cede1d9 00007fa12c1475e0 00007fa12c22cef1 00007fa12c20a183 00007fa12c1b8ac0 00007fa12a777bf4 00007fa12a777b82 00007fa12a777ab1
0504 13:56:28.222600 (+286483us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b5800000098. stack: 00007fa12409cc83 00007fa12409cb80 00007fa123c0d574 000000000050e094 000000000050e029 00007fa123f0c941 00007fa1240916c1 00007fa124091cf2 00007fa12cf07469 00007fa12c147488 00007fa12c20a240 00007fa12c1b8ac0 00007fa12a777bf4 00007fa12a777b82 00007fa12a777ab1 00007fa12a777a2c
0504 13:56:28.222697 (+ 97us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.222844 (+ 147us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:29.466540 (+1243696us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.551337 (+ 84797us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.551841 (+ 504us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4058,"spinlock_wait_cycles":43436800}
I20260504 13:56:29.560544 9806 rpcz_store.cc:276] 0504 13:56:27.318529 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.349009 (+ 30480us) service_pool.cc:224] Handling call
0504 13:56:28.016666 (+667657us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b5800000098. stack: 00007fa12409cc83 00007fa12409cb80 00007fa123c0d574 000000000050e094 000000000050e029 00007fa123f0c941 00007fa1240916c1 00007fa124091572 00007fa12cede1d9 00007fa12c1475e0 00007fa12c22cef1 00007fa12c20a183 00007fa12c1b8ac0 00007fa12a777bf4 00007fa12a777b82 00007fa12a777ab1
0504 13:56:28.234637 (+217971us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:28.234826 (+ 189us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:29.502237 (+1267411us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.546872 (+ 44635us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:29.547258 (+ 386us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9107,"spinlock_wait_cycles":8626432}
I20260504 13:56:29.553799 9931 tablet_bootstrap.cc:492] T 7eaa40eb4b1c425a9edfb7eaba953b65 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:29.561748 9931 ts_tablet_manager.cc:1403] T 7eaa40eb4b1c425a9edfb7eaba953b65 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 1.065s user 0.015s sys 0.046s
I20260504 13:56:29.558188 10743 rpcz_store.cc:276] 0504 13:56:28.029532 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.221815 (+192283us) service_pool.cc:224] Handling call
0504 13:56:28.540049 (+318234us) ts_tablet_manager.cc:1151] Deleting tablet bd8cd1caaeb249e3bafee7f1f0d4df2f
0504 13:56:28.540093 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:29.321752 (+781659us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.491317 (+169565us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.545312 (+ 53995us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":219514,"thread_start_us":226452,"threads_started":1}
I20260504 13:56:29.456146 9932 raft_consensus.cc:359] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.498421 10681 raft_consensus.cc:493] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:29.564425 10681 raft_consensus.cc:515] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.564805 10743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 210) took 1505 ms. Trace:
I20260504 13:56:29.565184 10743 rpcz_store.cc:276] 0504 13:56:28.059715 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.280478 (+220763us) service_pool.cc:224] Handling call
0504 13:56:29.564528 (+1284050us) ts_tablet_manager.cc:1151] Deleting tablet 1501455a0af348e38746d100987afdd0
0504 13:56:29.564575 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:29.564745 (+ 170us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1228252}
I20260504 13:56:29.566206 10681 leader_election.cc:290] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:29.566275 10743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 211) took 1506 ms. Trace:
I20260504 13:56:29.567651 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaa40eb4b1c425a9edfb7eaba953b65" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:29.568251 9537 raft_consensus.cc:2468] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:29.569434 9618 leader_election.cc:304] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:29.570262 9932 raft_consensus.cc:385] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:29.570585 9932 raft_consensus.cc:740] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:29.536594 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8702ce3ff1a845f3b70012855cd17efd" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:29.572332 10013 tablet_bootstrap.cc:654] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:29.572201 9932 consensus_queue.cc:260] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.544438 10683 raft_consensus.cc:493] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:29.575434 10683 raft_consensus.cc:515] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.576128 10611 raft_consensus.cc:493] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:29.576546 10611 raft_consensus.cc:515] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:29.571019 10681 raft_consensus.cc:2804] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:29.577334 10681 raft_consensus.cc:493] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:29.577564 10683 leader_election.cc:290] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:29.577666 10681 raft_consensus.cc:3060] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:29.578341 10611 leader_election.cc:290] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:29.579226 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274bb204041c4e1085395c7bd938aaf1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:29.580324 9474 leader_election.cc:343] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.580844 10788 raft_consensus.cc:493] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:29.581244 10788 raft_consensus.cc:515] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.581483 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0b9e47b0a4443c5830422b9d0553446" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:29.582985 10788 leader_election.cc:290] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:29.583637 9618 leader_election.cc:343] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.590482 10681 raft_consensus.cc:515] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.577636 9931 raft_consensus.cc:359] T 7eaa40eb4b1c425a9edfb7eaba953b65 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.592864 9931 raft_consensus.cc:385] T 7eaa40eb4b1c425a9edfb7eaba953b65 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:29.593129 9931 raft_consensus.cc:740] T 7eaa40eb4b1c425a9edfb7eaba953b65 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:29.594288 9806 tablet_service.cc:1511] Processing CreateTablet for tablet 75a43e11c8b34bd7bcddbf6e9b8f30dc (DEFAULT_TABLE table=default.Table_09255abca80f41dc895d81ccd89b9c7a [id=24485cf365ad4f9b89fa0f716fdb34cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.595229 9806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75a43e11c8b34bd7bcddbf6e9b8f30dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.595311 9339 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:29.598649 9797 tablet_service.cc:1511] Processing CreateTablet for tablet 57983fcecdb64a42884f797b4db57df0 (DEFAULT_TABLE table=default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.599644 9797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57983fcecdb64a42884f797b4db57df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.601997 9339 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 7eaa40eb4b1c425a9edfb7eaba953b65 (table default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e]). Expected version 1 got 0
I20260504 13:56:29.602720 9339 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 8db7c4732fa048f9950248f45d144c57 (table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62]). Expected version 1 got 0
I20260504 13:56:29.605326 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0b9e47b0a4443c5830422b9d0553446" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:29.606457 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaa40eb4b1c425a9edfb7eaba953b65" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:29.611696 9800 tablet_service.cc:1511] Processing CreateTablet for tablet 99fe335b131b45a587ec1cf526dd7f79 (DEFAULT_TABLE table=default.Table_655e2e8cf3544fbeb0ce33829bb2c8a0 [id=9db1e031520d405ea4989e30d5194911]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.612375 10013 tablet_bootstrap.cc:492] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:29.612723 9800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99fe335b131b45a587ec1cf526dd7f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.612774 10013 ts_tablet_manager.cc:1403] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.157s user 0.013s sys 0.010s
I20260504 13:56:29.521195 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3404ba04d3ec4a44a5e625536b941b58" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:29.605993 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274bb204041c4e1085395c7bd938aaf1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:29.621203 9686 raft_consensus.cc:3060] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:29.621307 9681 raft_consensus.cc:2468] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:29.625336 9931 consensus_queue.cc:260] T 7eaa40eb4b1c425a9edfb7eaba953b65 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.626832 9801 tablet_service.cc:1511] Processing CreateTablet for tablet 03e1d998c4314db3b3682a0510393749 (DEFAULT_TABLE table=default.Table_f00a5967b1b7422b830dcf11db2acbd8 [id=6a34d9af17c84984b479bbd10025fba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.627861 9801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03e1d998c4314db3b3682a0510393749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.633039 9832 raft_consensus.cc:2468] T 7eaa40eb4b1c425a9edfb7eaba953b65 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:29.627308 9336 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_60fe6ee7a2414e2aa8ea2d57ee4a827d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:29.607787 9932 ts_tablet_manager.cc:1434] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.285s user 0.009s sys 0.000s
W20260504 13:56:29.636092 9336 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60fe6ee7a2414e2aa8ea2d57ee4a827d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:29.628903 9338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_849bbe7902ca44969c5208828774fa06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:29.637295 9338 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_849bbe7902ca44969c5208828774fa06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:29.615110 9620 leader_election.cc:343] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.619377 10683 raft_consensus.cc:493] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:29.638105 10683 raft_consensus.cc:515] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.639758 9795 tablet_service.cc:1558] Processing DeleteTablet for tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 with delete_type TABLET_DATA_DELETED (replaced by tablet eef975650610423c88ce8e4cfb0b013b) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:29.639868 10683 leader_election.cc:290] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:29.641086 9620 leader_election.cc:304] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:29.641739 10683 raft_consensus.cc:2749] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:29.642465 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e4e583afc804212bece2f727411cc14" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:29.643436 10705 raft_consensus.cc:493] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:29.643846 10705 raft_consensus.cc:515] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.644140 9339 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 1c12ffa802b840b1937f6f796083c478 (table default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1]). Expected version 1 got 0
W20260504 13:56:29.644698 9618 leader_election.cc:343] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.645547 10705 leader_election.cc:290] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:29.648196 9473 leader_election.cc:304] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:29.652308 10705 raft_consensus.cc:2804] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:29.652629 9339 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]). Expected version 2 got 0
I20260504 13:56:29.652678 10705 raft_consensus.cc:493] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:29.653025 10705 raft_consensus.cc:3060] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:29.615332 10013 raft_consensus.cc:359] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.655157 10790 ts_tablet_manager.cc:1929] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:29.655646 10790 log.cc:1199] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:29.656360 10790 ts_tablet_manager.cc:1950] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:29.659273 10013 raft_consensus.cc:385] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:29.659574 10013 raft_consensus.cc:740] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:29.659327 10705 raft_consensus.cc:515] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:29.660166 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d63f665166466dba1e7e36c5e19506" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:29.661259 9474 leader_election.cc:343] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.661851 9932 tablet_bootstrap.cc:492] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:29.566615 10743 rpcz_store.cc:276] 0504 13:56:28.059954 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.281747 (+221793us) service_pool.cc:224] Handling call
0504 13:56:29.566029 (+1284282us) ts_tablet_manager.cc:1151] Deleting tablet c34877a81d504f46a2f680c5d5ddb4f4
0504 13:56:29.566070 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:29.566226 (+ 156us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1220152}
I20260504 13:56:29.664207 10013 consensus_queue.cc:260] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.665202 10743 tablet_replica.cc:333] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:29.666036 10743 raft_consensus.cc:2243] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:29.666313 10013 ts_tablet_manager.cc:1434] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.053s user 0.005s sys 0.000s
I20260504 13:56:29.666525 10743 raft_consensus.cc:2272] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:29.667914 9795 tablet_service.cc:1511] Processing CreateTablet for tablet df71dc4f2cf84cda8cfc534a0eea86f8 (DEFAULT_TABLE table=default.Table_1611aa9f4dba40e09dae4433e2d20154 [id=2fb2c56e2fb44b9aac476e06e36e5778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.668998 9795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df71dc4f2cf84cda8cfc534a0eea86f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.671878 9803 tablet_service.cc:1511] Processing CreateTablet for tablet 6dd139c347f7406aac340cc6ac994273 (DEFAULT_TABLE table=default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:29.672922 9803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd139c347f7406aac340cc6ac994273. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:29.674865 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet bd8cd1caaeb249e3bafee7f1f0d4df2f (table default.Table_47c9ed0b601146b687a576ec89c8b626 [id=b75b9e5edced4443944af4087b77e54d]) successfully deleted
W20260504 13:56:29.675582 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 1501455a0af348e38746d100987afdd0 because the tablet was not found. No further retry: Not found: Tablet not found: 1501455a0af348e38746d100987afdd0
I20260504 13:56:29.667148 10013 tablet_bootstrap.cc:492] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:29.668725 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3404ba04d3ec4a44a5e625536b941b58" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:29.679832 9537 raft_consensus.cc:3060] T 3404ba04d3ec4a44a5e625536b941b58 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:29.669973 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8702ce3ff1a845f3b70012855cd17efd" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:29.684207 9931 ts_tablet_manager.cc:1434] T 7eaa40eb4b1c425a9edfb7eaba953b65 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.122s user 0.005s sys 0.000s
I20260504 13:56:29.685084 9931 tablet_bootstrap.cc:492] T 9077fb9bb1544a2cbeb96f0d11c400db P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:29.688308 9537 raft_consensus.cc:2468] T 3404ba04d3ec4a44a5e625536b941b58 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:29.693732 9824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e4e583afc804212bece2f727411cc14" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:29.697583 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 (table default.Table_0134ffe7086440ccbcadd076d8fe5d01 [id=693e9b6ed89b4ab7b9d5dc7402e17f20]) successfully deleted
W20260504 13:56:29.698912 9620 leader_election.cc:343] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.701507 10013 tablet_bootstrap.cc:654] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:29.705345 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaa40eb4b1c425a9edfb7eaba953b65" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:29.705940 9537 raft_consensus.cc:3060] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:29.711920 10743 ts_tablet_manager.cc:1916] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:29.714313 9799 tablet_service.cc:1558] Processing DeleteTablet for tablet da61c91e10264fa49fe397d9a9b014bb with delete_type TABLET_DATA_DELETED (replaced by tablet 6323e1c10e7643f8a28f04e285f620e9) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:29.714278 9620 leader_election.cc:304] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:29.715076 10683 raft_consensus.cc:2749] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:29.716938 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d63f665166466dba1e7e36c5e19506" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:29.717495 9681 raft_consensus.cc:2468] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:29.718720 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274bb204041c4e1085395c7bd938aaf1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:29.719148 9681 raft_consensus.cc:3060] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:29.719982 9767 leader_election.cc:343] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Not found: Tablet not found: c34630d55f4a4e19b33ddf4c13995a5b
I20260504 13:56:29.722466 9767 leader_election.cc:304] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:29.723356 10787 raft_consensus.cc:2749] T 8702ce3ff1a845f3b70012855cd17efd P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:29.724489 9931 tablet_bootstrap.cc:654] T 9077fb9bb1544a2cbeb96f0d11c400db P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:29.724838 9767 leader_election.cc:304] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:29.727028 9681 raft_consensus.cc:2468] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:29.731559 10013 tablet_bootstrap.cc:492] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:29.731963 10013 ts_tablet_manager.cc:1403] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.004s
I20260504 13:56:29.735378 10787 raft_consensus.cc:2804] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:29.736052 10787 raft_consensus.cc:697] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:29.736893 10787 consensus_queue.cc:237] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:29.739687 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet c34877a81d504f46a2f680c5d5ddb4f4 because the tablet was not found. No further retry: Not found: Tablet not found: c34877a81d504f46a2f680c5d5ddb4f4
I20260504 13:56:29.739842 10391 ts_tablet_manager.cc:1929] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:29.742074 9686 raft_consensus.cc:2468] T 3404ba04d3ec4a44a5e625536b941b58 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:29.739485 10013 raft_consensus.cc:359] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.760876 9931 tablet_bootstrap.cc:492] T 9077fb9bb1544a2cbeb96f0d11c400db P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:29.761274 9931 ts_tablet_manager.cc:1403] T 9077fb9bb1544a2cbeb96f0d11c400db P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.076s user 0.007s sys 0.002s
I20260504 13:56:29.740300 10391 log.cc:1199] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/a1aba9a2a58b4e6b9e87f8ec80fcb6f1
I20260504 13:56:29.762802 10391 ts_tablet_manager.cc:1950] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:29.752451 10013 raft_consensus.cc:385] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:29.771390 10013 raft_consensus.cc:740] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:29.772022 10013 consensus_queue.cc:260] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:29.775556 9768 leader_election.cc:343] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: c34630d55f4a4e19b33ddf4c13995a5b
I20260504 13:56:29.775952 9768 leader_election.cc:304] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:29.779354 10013 ts_tablet_manager.cc:1434] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20260504 13:56:29.784749 10788 raft_consensus.cc:2749] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:29.787276 10681 leader_election.cc:290] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:29.789630 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eaa40eb4b1c425a9edfb7eaba953b65" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:29.790194 9832 raft_consensus.cc:3060] T 7eaa40eb4b1c425a9edfb7eaba953b65 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:29.791006 9931 raft_consensus.cc:359] T 9077fb9bb1544a2cbeb96f0d11c400db P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.795563 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 189) took 4223 ms. Trace:
I20260504 13:56:29.795867 10391 rpcz_store.cc:276] 0504 13:56:25.572472 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:25.576349 (+ 3877us) service_pool.cc:224] Handling call
0504 13:56:29.372373 (+3796024us) ts_tablet_manager.cc:1151] Deleting tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1
0504 13:56:29.372418 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:29.448903 (+ 76485us) spinlock_profiling.cc:243] Waited 32.5 ms on lock 0x7b2400002c78. stack: 00007f591ed51c83 00007f591ed51b80 00007f591e8c2574 000000000050e094 000000000050e029 00007f591ebc1941 00007f591ebd5e94 00007f591ebd3f9b 00007f591ec62d39 00007f591ec629c5 00007f591ec61f20 00007f5925bf9f45 00007f5925c13979 00007f5925c18113 00007f592626b915 00007f5926ec3cca
0504 13:56:29.648759 (+199856us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.739709 (+ 90950us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.795511 (+ 55802us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":74832640,"tablet-delete.queue_time_us":3795248}
I20260504 13:56:29.796767 9832 raft_consensus.cc:2468] T 7eaa40eb4b1c425a9edfb7eaba953b65 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:29.796958 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1 (table default.Table_7f9a1296bc424f7aa21625c670da0dd0 [id=a62d0dfdbe7741248a4d5d6614c712eb]) successfully deleted
I20260504 13:56:29.797343 10391 tablet_replica.cc:333] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:29.797998 9473 leader_election.cc:304] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:29.798916 9329 catalog_manager.cc:4982] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet da61c91e10264fa49fe397d9a9b014bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da61c91e10264fa49fe397d9a9b014bb already in progress: creating tablet
I20260504 13:56:29.800314 10013 tablet_bootstrap.cc:492] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:29.801914 9620 leader_election.cc:304] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:29.802709 10681 raft_consensus.cc:2804] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:29.803391 10681 raft_consensus.cc:697] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:29.804132 10681 consensus_queue.cc:237] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.817011 10431 raft_consensus.cc:2804] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:29.817437 10431 raft_consensus.cc:493] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:29.817790 10431 raft_consensus.cc:3060] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:29.791887 9931 raft_consensus.cc:385] T 9077fb9bb1544a2cbeb96f0d11c400db P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:29.825897 9931 raft_consensus.cc:740] T 9077fb9bb1544a2cbeb96f0d11c400db P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:29.826726 9931 consensus_queue.cc:260] T 9077fb9bb1544a2cbeb96f0d11c400db P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.829027 9931 ts_tablet_manager.cc:1434] T 9077fb9bb1544a2cbeb96f0d11c400db P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.068s user 0.008s sys 0.000s
I20260504 13:56:29.829886 9931 tablet_bootstrap.cc:492] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:29.846496 9474 consensus_peers.cc:597] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:29.847144 9474 consensus_peers.cc:597] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:29.847378 9537 raft_consensus.cc:2468] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:29.856410 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274bb204041c4e1085395c7bd938aaf1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:29.857666 9474 leader_election.cc:343] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.858374 10681 raft_consensus.cc:493] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:29.858996 10705 leader_election.cc:290] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:29.858803 10681 raft_consensus.cc:515] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.861518 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a470ddc5bcb4121bd2f5f02f6e6b945" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:29.861896 9932 tablet_bootstrap.cc:654] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:29.862195 10431 raft_consensus.cc:515] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:29.862831 9620 leader_election.cc:343] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.863615 10681 leader_election.cc:290] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:29.872535 10681 raft_consensus.cc:493] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:29.873433 10681 raft_consensus.cc:515] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.875780 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fae9274bb8240798f85b79504a227b3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:29.877013 9620 leader_election.cc:343] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.877591 10431 leader_election.cc:290] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:29.877816 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d63f665166466dba1e7e36c5e19506" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:29.878907 9474 leader_election.cc:343] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.880230 10391 raft_consensus.cc:2243] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:29.886821 10391 raft_consensus.cc:2272] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:29.889025 10391 ts_tablet_manager.cc:1916] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:29.892866 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a470ddc5bcb4121bd2f5f02f6e6b945" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:29.894228 10830 raft_consensus.cc:493] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:29.894726 10830 raft_consensus.cc:515] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.895378 10431 raft_consensus.cc:493] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:29.880918 9473 leader_election.cc:304] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:29.896064 10836 tablet.cc:1724] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:29.896584 10703 raft_consensus.cc:2804] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:29.897015 10703 raft_consensus.cc:697] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
W20260504 13:56:29.897439 9618 leader_election.cc:343] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.898345 10681 leader_election.cc:290] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:29.898281 9618 leader_election.cc:304] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:29.898300 10703 consensus_queue.cc:237] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:29.898903 10683 raft_consensus.cc:2749] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:29.900321 10830 leader_election.cc:290] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:29.897225 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "184fefa651fc411fb5107cc923c02535" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:29.908293 9474 leader_election.cc:343] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.912973 9680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d63f665166466dba1e7e36c5e19506" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:29.913513 9680 raft_consensus.cc:3060] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:29.914983 9331 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5cb30835150949f39f6e7e7563381763 consensus state has no leader
I20260504 13:56:29.921279 9680 raft_consensus.cc:2468] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:29.924211 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "184fefa651fc411fb5107cc923c02535" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:29.924819 9681 raft_consensus.cc:2468] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:29.927376 10431 raft_consensus.cc:515] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.932183 9932 tablet_bootstrap.cc:492] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:29.932593 9932 ts_tablet_manager.cc:1403] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.297s user 0.009s sys 0.002s
I20260504 13:56:29.936360 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fae9274bb8240798f85b79504a227b3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:29.938287 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68e92cc3da9b4bcfadb6fa26245f1c99" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:29.938395 10431 leader_election.cc:290] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:29.939383 9473 leader_election.cc:304] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:29.939777 9474 leader_election.cc:343] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.906451 10838 tablet.cc:1724] T 8db7c4732fa048f9950248f45d144c57 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260504 13:56:29.941637 9618 leader_election.cc:343] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:29.942076 9618 leader_election.cc:304] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:29.942334 10611 raft_consensus.cc:2804] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:29.942746 10611 raft_consensus.cc:697] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:29.943693 10683 raft_consensus.cc:2749] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:29.943662 10611 consensus_queue.cc:237] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.951864 9335 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 8db7c4732fa048f9950248f45d144c57 (table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62]). Expected version 1 got 0
I20260504 13:56:29.953148 9932 raft_consensus.cc:359] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.955897 9335 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 1fbe09b4078c4eebb52d7d00bc87371c (table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee]). Expected version 1 got 0
I20260504 13:56:29.957990 9932 raft_consensus.cc:385] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:29.958871 9932 raft_consensus.cc:740] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:29.960251 10013 tablet_bootstrap.cc:654] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:29.960784 9932 consensus_queue.cc:260] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:29.962936 10743 ts_tablet_manager.cc:1929] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:29.965265 9335 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:29.963117 10845 tablet.cc:1724] T 8db7c4732fa048f9950248f45d144c57 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:29.979436 9932 ts_tablet_manager.cc:1434] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.047s user 0.004s sys 0.002s
I20260504 13:56:29.991823 9932 tablet_bootstrap.cc:492] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:29.963354 10743 log.cc:1199] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/604f1ba5f71041a7877daff8972a994a
I20260504 13:56:29.994303 10743 ts_tablet_manager.cc:1950] T 604f1ba5f71041a7877daff8972a994a P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:29.996434 10743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 213) took 1936 ms. Trace:
I20260504 13:56:29.996753 10743 rpcz_store.cc:276] 0504 13:56:28.060423 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.352457 (+292034us) service_pool.cc:224] Handling call
0504 13:56:29.665046 (+1312589us) ts_tablet_manager.cc:1151] Deleting tablet 604f1ba5f71041a7877daff8972a994a
0504 13:56:29.665093 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:29.765070 (+ 99977us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.962817 (+197747us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:29.996380 (+ 33563us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1228350}
W20260504 13:56:29.997121 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 57797.7us
I20260504 13:56:29.997859 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 604f1ba5f71041a7877daff8972a994a (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]) successfully deleted
I20260504 13:56:29.997933 9473 leader_election.cc:304] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:29.999146 10611 raft_consensus.cc:2804] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:29.999457 10013 tablet_bootstrap.cc:492] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:29.999534 10611 raft_consensus.cc:493] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:29.999871 10013 ts_tablet_manager.cc:1403] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.200s user 0.013s sys 0.000s
I20260504 13:56:29.999984 10611 raft_consensus.cc:3060] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:30.002202 10013 raft_consensus.cc:359] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.002758 10013 raft_consensus.cc:385] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:30.008001 10611 raft_consensus.cc:515] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.011482 9680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68e92cc3da9b4bcfadb6fa26245f1c99" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:30.012658 9680 raft_consensus.cc:2468] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:30.015604 9931 tablet_bootstrap.cc:654] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:30.031472 10683 raft_consensus.cc:493] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:30.032395 10683 raft_consensus.cc:515] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.034061 10683 leader_election.cc:290] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:30.039579 10743 tablet_replica.cc:333] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:30.040556 10743 raft_consensus.cc:2243] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:30.043593 10743 raft_consensus.cc:2272] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:30.040910 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba4584755aae47e0b0f14c6c55474d18" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:29.993202 9335 catalog_manager.cc:5671] T 9f3905bfb30e4b03a7fcf0a2f57e2117 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:30.048784 9335 catalog_manager.cc:5671] T 8702ce3ff1a845f3b70012855cd17efd P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:30.050210 9335 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]). Expected version 2 got 0
W20260504 13:56:30.054524 9620 leader_election.cc:343] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:30.060014 9932 tablet_bootstrap.cc:654] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:30.010695 10830 consensus_queue.cc:1048] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:30.003005 10013 raft_consensus.cc:740] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:30.035946 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba4584755aae47e0b0f14c6c55474d18" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:30.068177 9618 leader_election.cc:343] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:30.068074 10013 consensus_queue.cc:260] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.068559 9618 leader_election.cc:304] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:30.068688 10743 ts_tablet_manager.cc:1916] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:30.069280 10611 leader_election.cc:290] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:30.070031 10683 raft_consensus.cc:2749] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:30.041081 10391 ts_tablet_manager.cc:1929] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:30.052178 9473 leader_election.cc:304] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:30.070570 10391 log.cc:1199] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/7c9030802900499a90c4dbaf5c6936a4
I20260504 13:56:30.070225 10013 ts_tablet_manager.cc:1434] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.070s user 0.005s sys 0.000s
I20260504 13:56:30.072333 10013 tablet_bootstrap.cc:492] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:30.072242 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "184fefa651fc411fb5107cc923c02535" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:30.072876 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "184fefa651fc411fb5107cc923c02535" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:30.073334 9686 raft_consensus.cc:3060] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:30.052786 10843 consensus_queue.cc:1048] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:30.075006 10703 raft_consensus.cc:2804] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260504 13:56:30.075347 9474 leader_election.cc:343] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:30.075356 10703 raft_consensus.cc:493] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:30.076166 10703 raft_consensus.cc:3060] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:30.079895 9686 raft_consensus.cc:2468] T 184fefa651fc411fb5107cc923c02535 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:30.080826 10391 ts_tablet_manager.cc:1950] T 7c9030802900499a90c4dbaf5c6936a4 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:30.085614 9473 leader_election.cc:304] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:30.090121 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 196) took 2991 ms. Trace:
I20260504 13:56:30.090716 10703 raft_consensus.cc:515] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.092571 10830 raft_consensus.cc:2804] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:30.093017 10830 raft_consensus.cc:697] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:30.091042 10391 rpcz_store.cc:276] 0504 13:56:27.098360 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.101236 (+ 2876us) service_pool.cc:224] Handling call
0504 13:56:29.797151 (+2695915us) ts_tablet_manager.cc:1151] Deleting tablet 7c9030802900499a90c4dbaf5c6936a4
0504 13:56:29.797203 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:29.926378 (+129175us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:30.040955 (+114577us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:30.090062 (+ 49107us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":81855,"tablet-delete.queue_time_us":2695396}
I20260504 13:56:30.093806 10830 consensus_queue.cc:237] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.110075 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 197) took 2986 ms. Trace:
I20260504 13:56:30.110596 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 7c9030802900499a90c4dbaf5c6936a4 (table default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]) successfully deleted
I20260504 13:56:30.114593 10743 ts_tablet_manager.cc:1929] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:30.119251 10391 rpcz_store.cc:276] 0504 13:56:27.124078 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.157364 (+ 33286us) service_pool.cc:224] Handling call
0504 13:56:30.109600 (+2952236us) ts_tablet_manager.cc:1151] Deleting tablet 21d6358b451d4b8d87a22e30e0780f44
0504 13:56:30.109654 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.110005 (+ 351us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2936154}
I20260504 13:56:30.120702 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 219) took 1900 ms. Trace:
W20260504 13:56:30.121228 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 21d6358b451d4b8d87a22e30e0780f44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21d6358b451d4b8d87a22e30e0780f44 already in progress: creating tablet
I20260504 13:56:30.122771 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68e92cc3da9b4bcfadb6fa26245f1c99" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:30.114970 10743 log.cc:1199] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/c34630d55f4a4e19b33ddf4c13995a5b
I20260504 13:56:30.131784 10743 ts_tablet_manager.cc:1950] T c34630d55f4a4e19b33ddf4c13995a5b P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:30.154439 9931 tablet_bootstrap.cc:492] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:30.154906 9931 ts_tablet_manager.cc:1403] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.325s user 0.008s sys 0.015s
I20260504 13:56:30.159809 9931 raft_consensus.cc:359] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:30.160357 9931 raft_consensus.cc:385] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:30.160578 9931 raft_consensus.cc:740] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:30.161437 9931 consensus_queue.cc:260] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:30.172838 10743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 214) took 2112 ms. Trace:
W20260504 13:56:30.124459 9474 leader_election.cc:343] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:30.179868 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 55440.7us
I20260504 13:56:30.180784 9932 tablet_bootstrap.cc:492] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:30.181181 9932 ts_tablet_manager.cc:1403] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.190s user 0.009s sys 0.001s
I20260504 13:56:30.182519 10703 leader_election.cc:290] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:30.182188 10743 rpcz_store.cc:276] 0504 13:56:28.060639 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.436756 (+376117us) service_pool.cc:224] Handling call
0504 13:56:30.039415 (+1602659us) ts_tablet_manager.cc:1151] Deleting tablet c34630d55f4a4e19b33ddf4c13995a5b
0504 13:56:30.039462 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.097039 (+ 57577us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:30.114465 (+ 17426us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:30.172775 (+ 58310us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1602062}
I20260504 13:56:30.183343 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet c34630d55f4a4e19b33ddf4c13995a5b (table default.Table_434bdc9867ad475a9719ba0c3760ef71 [id=db5cbb438d8f4da981e70b12713134f1]) successfully deleted
I20260504 13:56:30.183699 10743 tablet_replica.cc:333] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:30.178867 9931 ts_tablet_manager.cc:1434] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.024s user 0.002s sys 0.002s
I20260504 13:56:30.184264 9680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68e92cc3da9b4bcfadb6fa26245f1c99" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:30.184770 9680 raft_consensus.cc:3060] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:30.187479 10743 raft_consensus.cc:2243] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:30.188344 10743 pending_rounds.cc:70] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888: Trying to abort 1 pending ops.
I20260504 13:56:30.188573 10743 pending_rounds.cc:77] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290624401866752 op_type: NO_OP noop_request { }
I20260504 13:56:30.188946 10743 raft_consensus.cc:2889] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:30.189270 10743 raft_consensus.cc:2272] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:30.189961 9932 raft_consensus.cc:359] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.191632 9931 tablet_bootstrap.cc:492] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:30.191963 9932 raft_consensus.cc:385] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:30.192276 9932 raft_consensus.cc:740] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:30.192376 10703 raft_consensus.cc:493] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:30.192828 10703 raft_consensus.cc:515] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.193974 9932 consensus_queue.cc:260] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.195323 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41d41f8ba6dc4bb18116cb2b2dfabff0" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:30.196559 9474 leader_election.cc:343] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:30.197746 9932 ts_tablet_manager.cc:1434] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260504 13:56:30.198130 10703 leader_election.cc:290] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:30.199002 9473 leader_election.cc:343] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:30.195657 9680 raft_consensus.cc:2468] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:30.199380 9473 leader_election.cc:304] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:30.200351 9473 leader_election.cc:304] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:30.200906 10431 raft_consensus.cc:2749] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:30.201372 10431 raft_consensus.cc:2804] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:30.201720 10431 raft_consensus.cc:697] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:30.202332 10431 consensus_queue.cc:237] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.204057 10743 ts_tablet_manager.cc:1916] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:30.198575 9932 tablet_bootstrap.cc:492] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:30.217840 10013 tablet_bootstrap.cc:700] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent opening tablet: real 0.105s user 0.002s sys 0.000s
I20260504 13:56:30.218451 10013 tablet_bootstrap.cc:654] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:30.219141 10743 ts_tablet_manager.cc:1929] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:30.220103 10743 log.cc:1199] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/afad9329cf7745ba8168b2a93f998295
I20260504 13:56:30.220856 10743 ts_tablet_manager.cc:1950] T afad9329cf7745ba8168b2a93f998295 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:30.223116 10743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 215) took 2161 ms. Trace:
I20260504 13:56:30.223505 10743 rpcz_store.cc:276] 0504 13:56:28.061421 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.437400 (+375979us) service_pool.cc:224] Handling call
0504 13:56:30.183548 (+1746148us) ts_tablet_manager.cc:1151] Deleting tablet afad9329cf7745ba8168b2a93f998295
0504 13:56:30.183591 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.212252 (+ 28661us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:30.219030 (+ 6778us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:30.223060 (+ 4030us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1745625}
I20260504 13:56:30.224740 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet afad9329cf7745ba8168b2a93f998295 (table default.Table_cdee25533fe54da0a74b421714b0d7c4 [id=81df524d85774544807ba1902be1aae1]) successfully deleted
I20260504 13:56:30.224941 10743 tablet_replica.cc:333] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:30.226434 10743 raft_consensus.cc:2243] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:30.227403 10743 raft_consensus.cc:2272] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:30.184057 10391 rpcz_store.cc:276] 0504 13:56:28.220095 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.220204 (+ 109us) service_pool.cc:224] Handling call
0504 13:56:30.120454 (+1900250us) ts_tablet_manager.cc:1151] Deleting tablet 65052a5c28c147a385144317e21eea41
0504 13:56:30.120492 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.120644 (+ 152us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1899711}
W20260504 13:56:30.228989 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 65052a5c28c147a385144317e21eea41 because the tablet was not found. No further retry: Not found: Tablet not found: 65052a5c28c147a385144317e21eea41
I20260504 13:56:30.229481 10743 ts_tablet_manager.cc:1916] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:30.242524 9931 tablet_bootstrap.cc:654] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:30.245301 10391 tablet_replica.cc:333] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:30.249976 9932 tablet_bootstrap.cc:654] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:30.246112 10391 raft_consensus.cc:2243] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:30.253275 10391 raft_consensus.cc:2272] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:30.250850 10743 ts_tablet_manager.cc:1929] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:30.253628 10743 log.cc:1199] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/50e129980cd84f6ca22cbf2addb06230
I20260504 13:56:30.255160 10391 ts_tablet_manager.cc:1916] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:30.255299 10743 ts_tablet_manager.cc:1950] T 50e129980cd84f6ca22cbf2addb06230 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:30.257565 10743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 216) took 2195 ms. Trace:
I20260504 13:56:30.257917 10743 rpcz_store.cc:276] 0504 13:56:28.061759 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.437980 (+376221us) service_pool.cc:224] Handling call
0504 13:56:30.224792 (+1786812us) ts_tablet_manager.cc:1151] Deleting tablet 50e129980cd84f6ca22cbf2addb06230
0504 13:56:30.224835 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.237542 (+ 12707us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:30.250743 (+ 13201us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:30.257510 (+ 6767us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1786255}
I20260504 13:56:30.259130 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 50e129980cd84f6ca22cbf2addb06230 (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]) successfully deleted
I20260504 13:56:30.261335 10013 tablet_bootstrap.cc:492] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:30.261703 10013 ts_tablet_manager.cc:1403] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.190s user 0.009s sys 0.000s
I20260504 13:56:30.288365 10013 raft_consensus.cc:359] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.289196 10743 tablet_replica.cc:333] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:30.289956 10743 raft_consensus.cc:2243] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:30.291302 10845 tablet.cc:1724] T 1fbe09b4078c4eebb52d7d00bc87371c P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:30.292578 10743 raft_consensus.cc:2272] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:30.294139 10836 tablet.cc:1724] T 1fbe09b4078c4eebb52d7d00bc87371c P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:30.294667 10743 ts_tablet_manager.cc:1916] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:30.288919 10013 raft_consensus.cc:385] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:30.301831 10013 raft_consensus.cc:740] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:30.318421 10734 raft_consensus.cc:493] T 9077fb9bb1544a2cbeb96f0d11c400db P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:30.318861 10734 raft_consensus.cc:515] T 9077fb9bb1544a2cbeb96f0d11c400db P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.321224 10734 leader_election.cc:290] T 9077fb9bb1544a2cbeb96f0d11c400db P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:30.322152 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9077fb9bb1544a2cbeb96f0d11c400db" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:30.326350 10013 consensus_queue.cc:260] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.328411 10875 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.238.253:36337 every 8 connection(s)
I20260504 13:56:30.330364 10013 ts_tablet_manager.cc:1434] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.068s user 0.007s sys 0.000s
I20260504 13:56:30.331224 10013 tablet_bootstrap.cc:492] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:30.337119 9680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9077fb9bb1544a2cbeb96f0d11c400db" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:30.338375 9768 leader_election.cc:304] T 9077fb9bb1544a2cbeb96f0d11c400db P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:30.339533 10734 raft_consensus.cc:2749] T 9077fb9bb1544a2cbeb96f0d11c400db P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:30.344281 10703 raft_consensus.cc:493] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:30.345423 10838 tablet.cc:1724] T 1fbe09b4078c4eebb52d7d00bc87371c P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:30.345993 10703 raft_consensus.cc:515] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.347770 10086 rpc_server.cc:307] RPC server started. Bound to: 127.8.238.253:36337
I20260504 13:56:30.359913 10086 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/master-1/data/info.pb
I20260504 13:56:30.362689 9147 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu as pid 10086
I20260504 13:56:30.389670 10681 raft_consensus.cc:493] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:30.390148 10681 raft_consensus.cc:515] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.392179 10681 leader_election.cc:290] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:30.393736 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de6714ee86d64112baf13dd1b2b87582" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:30.395153 9620 leader_election.cc:343] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:30.397096 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de6714ee86d64112baf13dd1b2b87582" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:30.398351 9618 leader_election.cc:343] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:30.398682 9618 leader_election.cc:304] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:30.399729 9732 heartbeater.cc:344] Connected to a master server at 127.8.238.253:36337
I20260504 13:56:30.408080 10681 raft_consensus.cc:2749] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:30.422394 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c12ffa802b840b1937f6f796083c478" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:30.423430 9474 leader_election.cc:343] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:30.425400 10703 leader_election.cc:290] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:30.425360 9680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c12ffa802b840b1937f6f796083c478" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:30.426075 9680 raft_consensus.cc:2468] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:30.427018 9587 heartbeater.cc:344] Connected to a master server at 127.8.238.253:36337
I20260504 13:56:30.427196 9473 leader_election.cc:304] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:30.429394 9931 tablet_bootstrap.cc:492] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:30.429812 9931 ts_tablet_manager.cc:1403] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.245s user 0.009s sys 0.003s
I20260504 13:56:30.434934 9932 tablet_bootstrap.cc:492] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:30.439328 10703 raft_consensus.cc:2804] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:30.439729 10703 raft_consensus.cc:493] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:30.440097 10703 raft_consensus.cc:3060] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:30.443949 9932 ts_tablet_manager.cc:1403] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.246s user 0.011s sys 0.001s
I20260504 13:56:30.446419 9932 raft_consensus.cc:359] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.458113 10013 tablet_bootstrap.cc:654] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:30.460808 9931 raft_consensus.cc:359] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:30.461438 9931 raft_consensus.cc:385] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:30.461674 9931 raft_consensus.cc:740] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:30.474449 10703 raft_consensus.cc:515] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.447081 9932 raft_consensus.cc:385] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:30.478205 9932 raft_consensus.cc:740] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:30.487488 10734 raft_consensus.cc:493] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:30.487944 10734 raft_consensus.cc:515] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:30.489845 10734 leader_election.cc:290] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:30.492426 9767 leader_election.cc:343] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Not found: Tablet not found: 7c9030802900499a90c4dbaf5c6936a4
W20260504 13:56:30.493270 9768 leader_election.cc:343] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 7c9030802900499a90c4dbaf5c6936a4
I20260504 13:56:30.493636 9768 leader_election.cc:304] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:30.493521 9932 consensus_queue.cc:260] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.494414 9931 consensus_queue.cc:260] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:30.495307 10788 raft_consensus.cc:2749] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:30.500905 9932 ts_tablet_manager.cc:1434] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.057s user 0.005s sys 0.000s
I20260504 13:56:30.502158 9931 ts_tablet_manager.cc:1434] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.072s user 0.007s sys 0.000s
I20260504 13:56:30.507678 9932 tablet_bootstrap.cc:492] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:30.513782 9931 tablet_bootstrap.cc:492] T 8a985338790e4887af38c49ba013417f P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:30.528692 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c12ffa802b840b1937f6f796083c478" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:30.529838 9474 leader_election.cc:343] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:30.530609 10703 leader_election.cc:290] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:30.534816 9680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c12ffa802b840b1937f6f796083c478" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:30.535389 9680 raft_consensus.cc:3060] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:30.542192 9680 raft_consensus.cc:2468] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:30.543138 9473 leader_election.cc:304] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:30.544251 10703 raft_consensus.cc:2804] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:30.544711 10013 tablet_bootstrap.cc:492] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:30.544920 10703 raft_consensus.cc:697] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:30.545151 10013 ts_tablet_manager.cc:1403] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.214s user 0.006s sys 0.005s
I20260504 13:56:30.545751 10703 consensus_queue.cc:237] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.547549 10013 raft_consensus.cc:359] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.548120 10013 raft_consensus.cc:385] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:30.548353 10013 raft_consensus.cc:740] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:30.549117 10013 consensus_queue.cc:260] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.550902 10013 ts_tablet_manager.cc:1434] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260504 13:56:30.555684 10013 tablet_bootstrap.cc:492] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:30.560253 10391 ts_tablet_manager.cc:1929] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:30.560593 10391 log.cc:1199] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/50e129980cd84f6ca22cbf2addb06230
I20260504 13:56:30.561249 10391 ts_tablet_manager.cc:1950] T 50e129980cd84f6ca22cbf2addb06230 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:30.561905 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 220) took 2317 ms. Trace:
I20260504 13:56:30.562198 10391 rpcz_store.cc:276] 0504 13:56:28.244503 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.244693 (+ 190us) service_pool.cc:224] Handling call
0504 13:56:30.239328 (+1994635us) ts_tablet_manager.cc:1151] Deleting tablet 50e129980cd84f6ca22cbf2addb06230
0504 13:56:30.239999 (+ 671us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.466306 (+226307us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:30.560139 (+ 93833us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:30.561858 (+ 1719us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1994100}
I20260504 13:56:30.563316 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 50e129980cd84f6ca22cbf2addb06230 (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]) successfully deleted
I20260504 13:56:30.566668 10013 tablet_bootstrap.cc:654] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:30.573824 9932 tablet_bootstrap.cc:654] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:30.587539 9880 heartbeater.cc:344] Connected to a master server at 127.8.238.253:36337
I20260504 13:56:30.588281 10013 tablet_bootstrap.cc:492] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:30.588618 10013 ts_tablet_manager.cc:1403] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.001s
I20260504 13:56:30.590148 9931 tablet_bootstrap.cc:654] T 8a985338790e4887af38c49ba013417f P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:30.590866 10013 raft_consensus.cc:359] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.591698 10013 raft_consensus.cc:385] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:30.591954 10013 raft_consensus.cc:740] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:30.592573 10013 consensus_queue.cc:260] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.594784 10013 ts_tablet_manager.cc:1434] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260504 13:56:30.595629 10013 tablet_bootstrap.cc:492] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:30.596084 10743 ts_tablet_manager.cc:1929] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:30.596451 10743 log.cc:1199] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/030b35c3409143108c34ab1f0ee1b96d
I20260504 13:56:30.597182 10743 ts_tablet_manager.cc:1950] T 030b35c3409143108c34ab1f0ee1b96d P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:30.600596 9932 tablet_bootstrap.cc:492] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:30.600967 9932 ts_tablet_manager.cc:1403] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.093s user 0.008s sys 0.000s
I20260504 13:56:30.623482 9932 raft_consensus.cc:359] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.624096 9932 raft_consensus.cc:385] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:30.624357 9932 raft_consensus.cc:740] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:30.634387 10013 tablet_bootstrap.cc:654] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:30.636332 10743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 217) took 2574 ms. Trace:
I20260504 13:56:30.636668 10743 rpcz_store.cc:276] 0504 13:56:28.062033 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.438576 (+376543us) service_pool.cc:224] Handling call
0504 13:56:30.289048 (+1850472us) ts_tablet_manager.cc:1151] Deleting tablet 030b35c3409143108c34ab1f0ee1b96d
0504 13:56:30.289092 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.450489 (+161397us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:30.595954 (+145465us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:30.636268 (+ 40314us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1819595}
I20260504 13:56:30.610960 10881 sys_catalog.cc:263] Verifying existing consensus state
I20260504 13:56:30.638975 9880 heartbeater.cc:344] Connected to a master server at 127.8.238.253:36337
I20260504 13:56:30.637862 10743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 218) took 2575 ms. Trace:
I20260504 13:56:30.645504 10743 rpcz_store.cc:276] 0504 13:56:28.062306 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.439351 (+377045us) service_pool.cc:224] Handling call
0504 13:56:30.637629 (+2198278us) ts_tablet_manager.cc:1151] Deleting tablet baf07a970a7c4a35b3c849b06d6c88d0
0504 13:56:30.637668 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.637818 (+ 150us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2197582}
I20260504 13:56:30.637933 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 030b35c3409143108c34ab1f0ee1b96d (table default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]) successfully deleted
W20260504 13:56:30.648069 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet baf07a970a7c4a35b3c849b06d6c88d0 because the tablet was not found. No further retry: Not found: Tablet not found: baf07a970a7c4a35b3c849b06d6c88d0
I20260504 13:56:30.652098 10743 tablet_replica.cc:333] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:30.652985 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 221) took 2406 ms. Trace:
I20260504 13:56:30.653559 10391 rpcz_store.cc:276] 0504 13:56:28.246402 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.246511 (+ 109us) service_pool.cc:224] Handling call
0504 13:56:30.625701 (+2379190us) ts_tablet_manager.cc:1151] Deleting tablet 48b193e958b4414aad29dd2bcee1a61b
0504 13:56:30.652746 (+ 27045us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.652916 (+ 170us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2378669}
W20260504 13:56:30.654665 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 48b193e958b4414aad29dd2bcee1a61b because the tablet was not found. No further retry: Not found: Tablet not found: 48b193e958b4414aad29dd2bcee1a61b
I20260504 13:56:30.659134 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 222) took 2402 ms. Trace:
I20260504 13:56:30.659442 10391 rpcz_store.cc:276] 0504 13:56:28.257022 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.257141 (+ 119us) service_pool.cc:224] Handling call
0504 13:56:30.655416 (+2398275us) ts_tablet_manager.cc:1151] Deleting tablet 3c43ff1d759a40d68344f25f066416ac
0504 13:56:30.658941 (+ 3525us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.659086 (+ 145us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2397661}
W20260504 13:56:30.660508 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 3c43ff1d759a40d68344f25f066416ac because the tablet was not found. No further retry: Not found: Tablet not found: 3c43ff1d759a40d68344f25f066416ac
I20260504 13:56:30.661226 10788 consensus_queue.cc:1048] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:30.647310 9932 consensus_queue.cc:260] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.670984 10743 raft_consensus.cc:2243] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:30.671442 10743 pending_rounds.cc:70] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888: Trying to abort 1 pending ops.
I20260504 13:56:30.671635 10743 pending_rounds.cc:77] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282290634453331968 op_type: NO_OP noop_request { }
I20260504 13:56:30.671942 10743 raft_consensus.cc:2889] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260504 13:56:30.672240 10743 raft_consensus.cc:2272] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:30.672294 9931 tablet_bootstrap.cc:492] T 8a985338790e4887af38c49ba013417f P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:30.672674 9931 ts_tablet_manager.cc:1403] T 8a985338790e4887af38c49ba013417f P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.170s user 0.004s sys 0.006s
I20260504 13:56:30.675616 10881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7: Bootstrap starting.
I20260504 13:56:30.679828 10743 ts_tablet_manager.cc:1916] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:30.692265 9932 ts_tablet_manager.cc:1434] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.091s user 0.007s sys 0.000s
I20260504 13:56:30.692991 10681 raft_consensus.cc:493] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:30.674765 9931 raft_consensus.cc:359] T 8a985338790e4887af38c49ba013417f P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.693427 10681 raft_consensus.cc:515] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.694852 10431 raft_consensus.cc:493] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:30.693616 9931 raft_consensus.cc:385] T 8a985338790e4887af38c49ba013417f P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:30.696751 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e86027e3e74bc196e9a9ffc9fdf8a5" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:30.695159 10681 leader_election.cc:290] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:30.698552 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 223) took 2434 ms. Trace:
W20260504 13:56:30.699038 9620 leader_election.cc:343] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:30.698925 10391 rpcz_store.cc:276] 0504 13:56:28.264554 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.264665 (+ 111us) service_pool.cc:224] Handling call
0504 13:56:30.661118 (+2396453us) ts_tablet_manager.cc:1151] Deleting tablet 07a428049fdb4a68aae729400e2d9674
0504 13:56:30.661154 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.698499 (+ 37345us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2395904}
I20260504 13:56:30.702314 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 225) took 2410 ms. Trace:
W20260504 13:56:30.702746 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 07a428049fdb4a68aae729400e2d9674 because the tablet was not found. No further retry: Not found: Tablet not found: 07a428049fdb4a68aae729400e2d9674
I20260504 13:56:30.702638 10391 rpcz_store.cc:276] 0504 13:56:28.291659 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.291786 (+ 127us) service_pool.cc:224] Handling call
0504 13:56:30.702097 (+2410311us) ts_tablet_manager.cc:1151] Deleting tablet c34877a81d504f46a2f680c5d5ddb4f4
0504 13:56:30.702133 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.702256 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2409781}
W20260504 13:56:30.703655 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet c34877a81d504f46a2f680c5d5ddb4f4 because the tablet was not found. No further retry: Not found: Tablet not found: c34877a81d504f46a2f680c5d5ddb4f4
I20260504 13:56:30.696194 9931 raft_consensus.cc:740] T 8a985338790e4887af38c49ba013417f P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:30.705334 10391 tablet_replica.cc:333] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:30.707619 9931 consensus_queue.cc:260] T 8a985338790e4887af38c49ba013417f P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.695250 10431 raft_consensus.cc:515] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.693114 9932 tablet_bootstrap.cc:492] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:30.719872 9932 tablet_bootstrap.cc:654] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:30.724536 10743 ts_tablet_manager.cc:1929] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:30.727910 10391 raft_consensus.cc:2243] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:30.728359 10391 raft_consensus.cc:2272] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:30.728758 10431 leader_election.cc:290] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:30.729190 10431 consensus_queue.cc:1048] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:30.728281 9931 ts_tablet_manager.cc:1434] T 8a985338790e4887af38c49ba013417f P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.055s user 0.007s sys 0.000s
I20260504 13:56:30.753042 9932 tablet_bootstrap.cc:492] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:30.753513 9932 ts_tablet_manager.cc:1403] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.005s
I20260504 13:56:30.724870 10743 log.cc:1199] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/930078a10afa4dd987ad021cf9609514
I20260504 13:56:30.756429 10743 ts_tablet_manager.cc:1950] T 930078a10afa4dd987ad021cf9609514 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:30.756451 9932 raft_consensus.cc:359] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.758037 9932 raft_consensus.cc:385] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:30.758430 9932 raft_consensus.cc:740] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:30.760316 9932 consensus_queue.cc:260] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:30.765815 9932 ts_tablet_manager.cc:1434] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260504 13:56:30.766664 9932 tablet_bootstrap.cc:492] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:30.773056 10743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 219) took 2542 ms. Trace:
I20260504 13:56:30.773393 10743 rpcz_store.cc:276] 0504 13:56:28.214488 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.440113 (+225625us) service_pool.cc:224] Handling call
0504 13:56:30.651946 (+2211833us) ts_tablet_manager.cc:1151] Deleting tablet 930078a10afa4dd987ad021cf9609514
0504 13:56:30.651997 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.714712 (+ 62715us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:30.724413 (+ 9701us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:30.757157 (+ 32744us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2203854}
I20260504 13:56:30.774791 10743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 224) took 1135 ms. Trace:
I20260504 13:56:30.774748 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 930078a10afa4dd987ad021cf9609514 (table default.Table_0d2ce6c9ff35497b82dd58da3f913ecb [id=5a0c5354346746c186e36fe23406442c]) successfully deleted
I20260504 13:56:30.775050 10743 rpcz_store.cc:276] 0504 13:56:29.639501 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:29.639607 (+ 106us) service_pool.cc:224] Handling call
0504 13:56:30.774479 (+1134872us) ts_tablet_manager.cc:1151] Deleting tablet 41d41f8ba6dc4bb18116cb2b2dfabff0
0504 13:56:30.774517 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.774746 (+ 229us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1134266}
I20260504 13:56:30.776263 10743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56380 (request call id 227) took 1113 ms. Trace:
W20260504 13:56:30.776554 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 already in progress: creating tablet
I20260504 13:56:30.776552 10743 rpcz_store.cc:276] 0504 13:56:29.662450 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:29.662759 (+ 309us) service_pool.cc:224] Handling call
0504 13:56:30.775968 (+1113209us) ts_tablet_manager.cc:1151] Deleting tablet da61c91e10264fa49fe397d9a9b014bb
0504 13:56:30.776007 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.776220 (+ 213us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061238}
W20260504 13:56:30.777694 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet da61c91e10264fa49fe397d9a9b014bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da61c91e10264fa49fe397d9a9b014bb already in progress: creating tablet
I20260504 13:56:30.785598 10881 log.cc:826] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7: Log is configured to *not* fsync() on all Append() calls
I20260504 13:56:30.788563 9932 tablet_bootstrap.cc:654] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:30.794226 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e86027e3e74bc196e9a9ffc9fdf8a5" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:30.807646 9803 tablet_service.cc:1511] Processing CreateTablet for tablet 644170f17f1b45a19677d958b9745e16 (DEFAULT_TABLE table=default.Table_09fd612a3c4a4303b8f05530e7d49819 [id=eda14d7d5f7d4391a8c470677ffab4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:30.808661 9803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644170f17f1b45a19677d958b9745e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:30.809674 9340 catalog_manager.cc:5671] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:30.814198 9340 catalog_manager.cc:5671] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:30.814905 9795 tablet_service.cc:1511] Processing CreateTablet for tablet 9f80ce19e8f9498b98952a371476f185 (DEFAULT_TABLE table=default.Table_2bed5ea8f43149e8bc3b21cd9d1b3846 [id=15994ccc978545058e324152f70c8c3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:30.815946 9795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f80ce19e8f9498b98952a371476f185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:30.821662 9340 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 1fbe09b4078c4eebb52d7d00bc87371c (table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee]). Expected version 1 got 0
I20260504 13:56:30.823258 9340 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 8db7c4732fa048f9950248f45d144c57 (table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62]). Expected version 1 got 0
I20260504 13:56:30.823400 9931 tablet_bootstrap.cc:492] T 184fefa651fc411fb5107cc923c02535 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:30.828883 9931 tablet_bootstrap.cc:654] T 184fefa651fc411fb5107cc923c02535 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:30.829852 9340 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:30.832463 9340 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]). Expected version 2 got 0
I20260504 13:56:30.834019 9797 tablet_service.cc:1511] Processing CreateTablet for tablet 7989338bc4bf4bad8fb8fe37587d0a16 (DEFAULT_TABLE table=default.Table_10e92469b9cb4e9b965d0a10cfeecca8 [id=99da5017cead4ddeaa3441064b1ed716]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:30.836979 9797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7989338bc4bf4bad8fb8fe37587d0a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:30.851052 9803 tablet_service.cc:1511] Processing CreateTablet for tablet 8d6db45d2e5c41a3b30cede3301694d3 (DEFAULT_TABLE table=default.Table_862ae682dfb44737a72e7c5df628a957 [id=bbef146162cb4c30a3ba42ca2a36fe9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:30.852224 9803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d6db45d2e5c41a3b30cede3301694d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:30.862097 10391 ts_tablet_manager.cc:1916] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:30.868700 9512 tablet_service.cc:1511] Processing CreateTablet for tablet 9f80ce19e8f9498b98952a371476f185 (DEFAULT_TABLE table=default.Table_2bed5ea8f43149e8bc3b21cd9d1b3846 [id=15994ccc978545058e324152f70c8c3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:30.870000 9512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f80ce19e8f9498b98952a371476f185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:30.872381 9507 tablet_service.cc:1511] Processing CreateTablet for tablet 644170f17f1b45a19677d958b9745e16 (DEFAULT_TABLE table=default.Table_09fd612a3c4a4303b8f05530e7d49819 [id=eda14d7d5f7d4391a8c470677ffab4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:30.873744 9507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644170f17f1b45a19677d958b9745e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:30.856000 9799 tablet_service.cc:1511] Processing CreateTablet for tablet 37b1664312d347288680d97b62c2aecb (DEFAULT_TABLE table=default.Table_861ff05bb9e144ea85a3fb7d75f68749 [id=c4b5a9431b8b4d33b41fa27a357e1f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:30.881495 9799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37b1664312d347288680d97b62c2aecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:30.883642 10788 raft_consensus.cc:493] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:30.884076 10788 raft_consensus.cc:515] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:30.885900 10788 leader_election.cc:290] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:30.905500 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 205209us
I20260504 13:56:30.918160 10881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 13:56:30.919193 10881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7: Bootstrap complete.
I20260504 13:56:30.923519 9806 tablet_service.cc:1511] Processing CreateTablet for tablet ddd30e9259a04d59b5c1bb82aabf7952 (DEFAULT_TABLE table=default.Table_512ae07c2df74cb99bb77006d985f2da [id=09959f81c69c49778a6ee33dc1e85aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:30.924453 9806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddd30e9259a04d59b5c1bb82aabf7952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:30.939620 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 1fbe09b4078c4eebb52d7d00bc87371c (table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee]). Expected version 1 got 0
I20260504 13:56:30.943506 9587 heartbeater.cc:344] Connected to a master server at 127.8.238.253:36337
I20260504 13:56:30.944698 10431 consensus_queue.cc:1048] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:30.961022 9801 tablet_service.cc:1511] Processing CreateTablet for tablet eef975650610423c88ce8e4cfb0b013b (DEFAULT_TABLE table=default.Table_0134ffe7086440ccbcadd076d8fe5d01 [id=693e9b6ed89b4ab7b9d5dc7402e17f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:30.962182 9801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef975650610423c88ce8e4cfb0b013b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:30.965214 9800 tablet_service.cc:1511] Processing CreateTablet for tablet cf11f17e95dc411294c755e17c57811e (DEFAULT_TABLE table=default.Table_2dc0381bd6c84ab59d5bb422de89c0e2 [id=ed756f69850642188b881bd3c680a622]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:30.966277 9800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf11f17e95dc411294c755e17c57811e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:30.942315 9329 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 1c12ffa802b840b1937f6f796083c478 (table default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c12ffa802b840b1937f6f796083c478 consensus state has no leader
I20260504 13:56:30.969322 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 274bb204041c4e1085395c7bd938aaf1 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]). Expected version 4 got 0
I20260504 13:56:30.981648 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 7eaa40eb4b1c425a9edfb7eaba953b65 (table default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e]). Expected version 1 got 0
I20260504 13:56:30.988292 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 8db7c4732fa048f9950248f45d144c57 (table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62]). Expected version 1 got 0
I20260504 13:56:30.992233 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38]). Expected version 1 got 0
I20260504 13:56:30.994081 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]). Expected version 2 got 0
I20260504 13:56:30.995690 9806 tablet_service.cc:1511] Processing CreateTablet for tablet e29a7365caf04506b15089f10e6c1677 (DEFAULT_TABLE table=default.Table_6b0124eccb02454b9195b8ddaae5bd38 [id=d47a2c6a9cb34ba4ade10fc23913bf7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:30.996800 9806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e29a7365caf04506b15089f10e6c1677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:30.997422 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 1c12ffa802b840b1937f6f796083c478 (table default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1]). Expected version 1 got 0
I20260504 13:56:31.004858 9337 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:31.032167 9338 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 8db7c4732fa048f9950248f45d144c57 (table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62]). Expected version 1 got 0
I20260504 13:56:31.037748 9338 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 1fbe09b4078c4eebb52d7d00bc87371c (table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee]). Expected version 1 got 0
I20260504 13:56:31.038537 9338 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 274bb204041c4e1085395c7bd938aaf1 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]). Expected version 4 got 0
I20260504 13:56:31.039433 9338 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:31.018764 10881 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:31.012480 9799 tablet_service.cc:1511] Processing CreateTablet for tablet 2c09e9076214405eaa8846fe08158bb3 (DEFAULT_TABLE table=default.Table_6e41cb79b95848ba8c7b1f05f961c7f3 [id=b91fe7c5c3844c109756842f93098699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.012475 9803 tablet_service.cc:1511] Processing CreateTablet for tablet 6323e1c10e7643f8a28f04e285f620e9 (DEFAULT_TABLE table=default.Table_999e37d0dab54ca68c0fa66f74582109 [id=e37fd7e36e604a60832acf303f0ecb7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.040542 10881 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6caedf407d13459e8fd477e8b60b7dc7, State: Initialized, Role: FOLLOWER
I20260504 13:56:31.041952 9803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6323e1c10e7643f8a28f04e285f620e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.041579 10881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:31.044723 9799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c09e9076214405eaa8846fe08158bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.046895 10391 ts_tablet_manager.cc:1929] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:31.057145 10904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:31.057914 10904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:56:31.060757 10881 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260504 13:56:30.814270 9657 tablet_service.cc:1511] Processing CreateTablet for tablet 644170f17f1b45a19677d958b9745e16 (DEFAULT_TABLE table=default.Table_09fd612a3c4a4303b8f05530e7d49819 [id=eda14d7d5f7d4391a8c470677ffab4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.072005 9657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644170f17f1b45a19677d958b9745e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:30.819278 9659 tablet_service.cc:1511] Processing CreateTablet for tablet 9f80ce19e8f9498b98952a371476f185 (DEFAULT_TABLE table=default.Table_2bed5ea8f43149e8bc3b21cd9d1b3846 [id=15994ccc978545058e324152f70c8c3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.075973 9659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f80ce19e8f9498b98952a371476f185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.062985 9338 catalog_manager.cc:5671] T 9077fb9bb1544a2cbeb96f0d11c400db P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:30.837886 9655 tablet_service.cc:1511] Processing CreateTablet for tablet 7989338bc4bf4bad8fb8fe37587d0a16 (DEFAULT_TABLE table=default.Table_10e92469b9cb4e9b965d0a10cfeecca8 [id=99da5017cead4ddeaa3441064b1ed716]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.079159 9338 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]). Expected version 2 got 0
I20260504 13:56:31.080011 9655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7989338bc4bf4bad8fb8fe37587d0a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.080063 9338 catalog_manager.cc:5671] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:31.082126 9338 catalog_manager.cc:5671] T 8a985338790e4887af38c49ba013417f P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:30.855345 9665 tablet_service.cc:1511] Processing CreateTablet for tablet 8d6db45d2e5c41a3b30cede3301694d3 (DEFAULT_TABLE table=default.Table_862ae682dfb44737a72e7c5df628a957 [id=bbef146162cb4c30a3ba42ca2a36fe9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.083753 9665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d6db45d2e5c41a3b30cede3301694d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:30.888224 9680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60d3cc59ffe947fbb57b7d3301ccbac1" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:30.907016 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bae90898739b4303af0608e39e8c0367" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:30.922080 9666 tablet_service.cc:1511] Processing CreateTablet for tablet 37b1664312d347288680d97b62c2aecb (DEFAULT_TABLE table=default.Table_861ff05bb9e144ea85a3fb7d75f68749 [id=c4b5a9431b8b4d33b41fa27a357e1f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.086807 9682 raft_consensus.cc:2468] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:31.087141 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37b1664312d347288680d97b62c2aecb. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:31.087277 9341 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60d3cc59ffe947fbb57b7d3301ccbac1 consensus state has no leader
I20260504 13:56:31.087774 9341 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35768 (request call id 259) took 5385 ms. Trace:
I20260504 13:56:31.088142 9341 rpcz_store.cc:276] 0504 13:56:25.702387 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.149578 (+2447191us) service_pool.cc:224] Handling call
0504 13:56:28.191250 (+ 41672us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:28.227589 (+ 36339us) catalog_manager.cc:3725] Apply alter schema
0504 13:56:28.228134 (+ 545us) catalog_manager.cc:3757] Acquired catalog manager lock
0504 13:56:28.228375 (+ 241us) catalog_manager.cc:3915] Updating metadata on disk
0504 13:56:28.285568 (+ 57193us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:28.285592 (+ 24us) write_op.cc:454] Released schema lock
0504 13:56:28.287293 (+ 1701us) catalog_manager.cc:3946] Committing alterations to in-memory state
0504 13:56:31.087709 (+2800416us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:28.267436 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:28.267882 (+ 446us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:28.267915 (+ 33us) write_op.cc:435] Acquired schema lock
0504 13:56:28.267931 (+ 16us) tablet.cc:662] Decoding operations
0504 13:56:28.268209 (+ 278us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:28.268266 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:28.268296 (+ 30us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:28.268467 (+ 171us) tablet.cc:701] Row locks acquired
0504 13:56:28.268483 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:28.268598 (+ 115us) write_op.cc:270] Start()
0504 13:56:28.268693 (+ 95us) write_op.cc:276] Timestamp: P: 1777902988268499 usec, L: 0
0504 13:56:28.268720 (+ 27us) op_driver.cc:348] REPLICATION: starting
0504 13:56:28.269364 (+ 644us) log.cc:844] Serialized 405 byte log entry
0504 13:56:28.279126 (+ 9762us) op_driver.cc:464] REPLICATION: finished
0504 13:56:28.279509 (+ 383us) write_op.cc:301] APPLY: starting
0504 13:56:28.279566 (+ 57us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:28.279667 (+ 101us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:28.279771 (+ 104us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:28.279975 (+ 204us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:28.280139 (+ 164us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:28.280188 (+ 49us) write_op.cc:312] APPLY: finished
0504 13:56:28.280385 (+ 197us) log.cc:844] Serialized 23 byte log entry
0504 13:56:28.280514 (+ 129us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:28.280570 (+ 56us) write_op.cc:454] Released schema lock
0504 13:56:28.280608 (+ 38us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":198,"apply.run_cpu_time_us":1294,"apply.run_wall_time_us":1294,"num_ops":1,"prepare.queue_time_us":37614,"prepare.run_cpu_time_us":2334,"prepare.run_wall_time_us":2334,"raft.queue_time_us":1699,"raft.run_cpu_time_us":773,"raft.run_wall_time_us":5963,"replication_time_us":10304}]]}
I20260504 13:56:30.976929 9656 tablet_service.cc:1511] Processing CreateTablet for tablet ddd30e9259a04d59b5c1bb82aabf7952 (DEFAULT_TABLE table=default.Table_512ae07c2df74cb99bb77006d985f2da [id=09959f81c69c49778a6ee33dc1e85aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.092365 9656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddd30e9259a04d59b5c1bb82aabf7952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:30.979758 9661 tablet_service.cc:1511] Processing CreateTablet for tablet eef975650610423c88ce8e4cfb0b013b (DEFAULT_TABLE table=default.Table_0134ffe7086440ccbcadd076d8fe5d01 [id=693e9b6ed89b4ab7b9d5dc7402e17f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:30.981530 9664 tablet_service.cc:1511] Processing CreateTablet for tablet cf11f17e95dc411294c755e17c57811e (DEFAULT_TABLE table=default.Table_2dc0381bd6c84ab59d5bb422de89c0e2 [id=ed756f69850642188b881bd3c680a622]), partition=RANGE (key) PARTITION UNBOUNDED
W20260504 13:56:31.092415 9341 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.8.238.194:38543 (request call id 40) took 802 ms (client timeout 1000 ms). Trace:
W20260504 13:56:31.092686 9341 rpcz_store.cc:269] 0504 13:56:30.290320 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:31.092222 (+801902us) service_pool.cc:224] Handling call
0504 13:56:31.092355 (+ 133us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:30.984521 10681 consensus_queue.cc:1048] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:31.093147 9664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf11f17e95dc411294c755e17c57811e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.093536 9731 heartbeater.cc:344] Connected to a master server at 127.8.238.252:44781
I20260504 13:56:30.994640 9663 tablet_service.cc:1511] Processing CreateTablet for tablet e29a7365caf04506b15089f10e6c1677 (DEFAULT_TABLE table=default.Table_6b0124eccb02454b9195b8ddaae5bd38 [id=d47a2c6a9cb34ba4ade10fc23913bf7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.095726 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e29a7365caf04506b15089f10e6c1677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.007879 9660 tablet_service.cc:1511] Processing CreateTablet for tablet 6323e1c10e7643f8a28f04e285f620e9 (DEFAULT_TABLE table=default.Table_999e37d0dab54ca68c0fa66f74582109 [id=e37fd7e36e604a60832acf303f0ecb7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.097024 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6323e1c10e7643f8a28f04e285f620e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.012408 9662 tablet_service.cc:1511] Processing CreateTablet for tablet 2c09e9076214405eaa8846fe08158bb3 (DEFAULT_TABLE table=default.Table_6e41cb79b95848ba8c7b1f05f961c7f3 [id=b91fe7c5c3844c109756842f93098699]), partition=RANGE (key) PARTITION UNBOUNDED
W20260504 13:56:30.796530 9618 leader_election.cc:343] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:31.102402 9618 leader_election.cc:304] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:31.103677 10681 raft_consensus.cc:2749] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:31.100201 9662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c09e9076214405eaa8846fe08158bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.104929 9661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef975650610423c88ce8e4cfb0b013b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:31.103135 9618 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 306621us
I20260504 13:56:31.118686 10830 raft_consensus.cc:493] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:31.119120 10830 raft_consensus.cc:515] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.123395 10830 leader_election.cc:290] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:31.136853 10681 consensus_queue.cc:1048] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:31.149119 9336 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:31.150347 9336 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]). Expected version 2 got 0
I20260504 13:56:31.151391 9336 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 8db7c4732fa048f9950248f45d144c57 (table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62]). Expected version 1 got 0
I20260504 13:56:31.152187 9336 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 1fbe09b4078c4eebb52d7d00bc87371c (table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee]). Expected version 1 got 0
I20260504 13:56:31.171047 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ebc96408d64c2da7dc6c4fc158995c" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:31.172391 9682 raft_consensus.cc:2468] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:31.176137 9931 tablet_bootstrap.cc:492] T 184fefa651fc411fb5107cc923c02535 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:31.176542 9931 ts_tablet_manager.cc:1403] T 184fefa651fc411fb5107cc923c02535 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.401s user 0.035s sys 0.028s
I20260504 13:56:31.178795 9931 raft_consensus.cc:359] T 184fefa651fc411fb5107cc923c02535 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.179545 9931 raft_consensus.cc:385] T 184fefa651fc411fb5107cc923c02535 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:31.179811 9931 raft_consensus.cc:740] T 184fefa651fc411fb5107cc923c02535 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:31.180365 9931 consensus_queue.cc:260] T 184fefa651fc411fb5107cc923c02535 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.182238 9931 ts_tablet_manager.cc:1434] T 184fefa651fc411fb5107cc923c02535 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260504 13:56:31.198611 10788 consensus_queue.cc:1048] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:31.202476 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bae90898739b4303af0608e39e8c0367" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:31.183058 9931 tablet_bootstrap.cc:492] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:31.211632 9509 tablet_service.cc:1511] Processing CreateTablet for tablet 8d6db45d2e5c41a3b30cede3301694d3 (DEFAULT_TABLE table=default.Table_862ae682dfb44737a72e7c5df628a957 [id=bbef146162cb4c30a3ba42ca2a36fe9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.212561 9509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d6db45d2e5c41a3b30cede3301694d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.228569 9931 tablet_bootstrap.cc:654] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:31.246443 9732 heartbeater.cc:344] Connected to a master server at 127.8.238.253:36337
I20260504 13:56:31.246953 9473 leader_election.cc:304] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:31.256433 9931 tablet_bootstrap.cc:492] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:31.256814 9931 ts_tablet_manager.cc:1403] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.074s user 0.008s sys 0.000s
I20260504 13:56:31.268402 9831 raft_consensus.cc:3060] T 9077fb9bb1544a2cbeb96f0d11c400db P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:31.259019 9931 raft_consensus.cc:359] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:31.271492 9931 raft_consensus.cc:385] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:31.271747 9931 raft_consensus.cc:740] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:31.273846 9931 consensus_queue.cc:260] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:31.275053 9341 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:31.276224 9341 catalog_manager.cc:5671] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:31.283022 9341 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 7eaa40eb4b1c425a9edfb7eaba953b65 (table default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e]). Expected version 1 got 0
I20260504 13:56:31.283700 9797 tablet_service.cc:1511] Processing CreateTablet for tablet e0e70772b7ce4f27a941918aaf66a865 (DEFAULT_TABLE table=default.Table_60fe6ee7a2414e2aa8ea2d57ee4a827d [id=1de44a1bbf06492084cfd70cf540b213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.284773 9797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e70772b7ce4f27a941918aaf66a865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.285058 9931 ts_tablet_manager.cc:1434] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260504 13:56:31.286713 9341 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 60d3cc59ffe947fbb57b7d3301ccbac1 (table default.Table_3bc454a7435a4553be5a9bbcfec24eb3 [id=994b65b95f66440283f852cd9dd9ae0c]). Expected version 1 got 0
I20260504 13:56:31.288478 9341 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]). Expected version 2 got 0
I20260504 13:56:31.291366 9343 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet de639fda874248f0853e74ee60aaf758
I20260504 13:56:31.291590 9931 tablet_bootstrap.cc:492] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:31.292222 9795 tablet_service.cc:1511] Processing CreateTablet for tablet bd635a863f7a41d087947f619e41f95c (DEFAULT_TABLE table=default.Table_849bbe7902ca44969c5208828774fa06 [id=c146e84e0e844deda9af40562f980047]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.292563 9343 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35768 (request call id 261) took 5589 ms. Trace:
I20260504 13:56:31.293243 9795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd635a863f7a41d087947f619e41f95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.296607 9660 tablet_service.cc:1511] Processing CreateTablet for tablet e0e70772b7ce4f27a941918aaf66a865 (DEFAULT_TABLE table=default.Table_60fe6ee7a2414e2aa8ea2d57ee4a827d [id=1de44a1bbf06492084cfd70cf540b213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.297708 9664 tablet_service.cc:1511] Processing CreateTablet for tablet bd635a863f7a41d087947f619e41f95c (DEFAULT_TABLE table=default.Table_849bbe7902ca44969c5208828774fa06 [id=c146e84e0e844deda9af40562f980047]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.249531 9517 tablet_service.cc:1511] Processing CreateTablet for tablet 37b1664312d347288680d97b62c2aecb (DEFAULT_TABLE table=default.Table_861ff05bb9e144ea85a3fb7d75f68749 [id=c4b5a9431b8b4d33b41fa27a357e1f46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.299525 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37b1664312d347288680d97b62c2aecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.292826 9343 rpcz_store.cc:276] 0504 13:56:25.702860 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.158831 (+2455971us) service_pool.cc:224] Handling call
0504 13:56:28.225808 (+ 66977us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:28.226411 (+ 603us) catalog_manager.cc:2815] Modifying in-memory table state
0504 13:56:28.226568 (+ 157us) catalog_manager.cc:2824] Locking tablets
0504 13:56:28.249015 (+ 22447us) catalog_manager.cc:2838] Removing table and tablets from system table
0504 13:56:28.345508 (+ 96493us) catalog_manager.cc:2856] Removing table from by-name map
0504 13:56:28.345717 (+ 209us) catalog_manager.cc:2872] Committing in-memory state
0504 13:56:31.291290 (+2945573us) catalog_manager.cc:2876] Aborting table tasks
0504 13:56:31.292495 (+ 1205us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:28.306879 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:28.308193 (+ 1314us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:28.308218 (+ 25us) write_op.cc:435] Acquired schema lock
0504 13:56:28.308233 (+ 15us) tablet.cc:662] Decoding operations
0504 13:56:28.308533 (+ 300us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:28.308568 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:28.308595 (+ 27us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:28.308899 (+ 304us) tablet.cc:701] Row locks acquired
0504 13:56:28.308916 (+ 17us) write_op.cc:260] PREPARE: finished
0504 13:56:28.309011 (+ 95us) write_op.cc:270] Start()
0504 13:56:28.309085 (+ 74us) write_op.cc:276] Timestamp: P: 1777902988308922 usec, L: 0
0504 13:56:28.309102 (+ 17us) op_driver.cc:348] REPLICATION: starting
0504 13:56:28.309561 (+ 459us) log.cc:844] Serialized 771 byte log entry
0504 13:56:28.322984 (+ 13423us) op_driver.cc:464] REPLICATION: finished
0504 13:56:28.325592 (+ 2608us) write_op.cc:301] APPLY: starting
0504 13:56:28.325635 (+ 43us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:28.325725 (+ 90us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:28.325813 (+ 88us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:28.326057 (+ 244us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:28.326171 (+ 114us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:28.326212 (+ 41us) write_op.cc:312] APPLY: finished
0504 13:56:28.327205 (+ 993us) log.cc:844] Serialized 29 byte log entry
0504 13:56:28.327366 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:28.327433 (+ 67us) write_op.cc:454] Released schema lock
0504 13:56:28.327466 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2493,"apply.run_cpu_time_us":1212,"apply.run_wall_time_us":2047,"mutex_wait_us":53629,"num_ops":2,"prepare.queue_time_us":56124,"prepare.run_cpu_time_us":2124,"prepare.run_wall_time_us":28971,"raft.queue_time_us":1408,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":9,"replication_time_us":13802,"thread_start_us":49631,"threads_started":1}]]}
I20260504 13:56:31.308055 9655 tablet_service.cc:1558] Processing DeleteTablet for tablet de639fda874248f0853e74ee60aaf758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:28 UTC) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:31.312285 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 65415.3us
I20260504 13:56:31.312666 9505 tablet_service.cc:1511] Processing CreateTablet for tablet cf11f17e95dc411294c755e17c57811e (DEFAULT_TABLE table=default.Table_2dc0381bd6c84ab59d5bb422de89c0e2 [id=ed756f69850642188b881bd3c680a622]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.313695 9505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf11f17e95dc411294c755e17c57811e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.278625 9516 tablet_service.cc:1511] Processing CreateTablet for tablet ddd30e9259a04d59b5c1bb82aabf7952 (DEFAULT_TABLE table=default.Table_512ae07c2df74cb99bb77006d985f2da [id=09959f81c69c49778a6ee33dc1e85aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.281721 9508 tablet_service.cc:1511] Processing CreateTablet for tablet eef975650610423c88ce8e4cfb0b013b (DEFAULT_TABLE table=default.Table_0134ffe7086440ccbcadd076d8fe5d01 [id=693e9b6ed89b4ab7b9d5dc7402e17f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.312388 10890 raft_consensus.cc:2804] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:31.315403 10890 raft_consensus.cc:493] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:31.315732 10890 raft_consensus.cc:3060] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:31.315732 9516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddd30e9259a04d59b5c1bb82aabf7952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.276894 10013 tablet_bootstrap.cc:492] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:31.317691 10013 ts_tablet_manager.cc:1403] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.722s user 0.015s sys 0.004s
I20260504 13:56:31.319796 9508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef975650610423c88ce8e4cfb0b013b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.321949 9510 tablet_service.cc:1511] Processing CreateTablet for tablet e29a7365caf04506b15089f10e6c1677 (DEFAULT_TABLE table=default.Table_6b0124eccb02454b9195b8ddaae5bd38 [id=d47a2c6a9cb34ba4ade10fc23913bf7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.322878 9510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e29a7365caf04506b15089f10e6c1677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.326182 9506 tablet_service.cc:1511] Processing CreateTablet for tablet 6323e1c10e7643f8a28f04e285f620e9 (DEFAULT_TABLE table=default.Table_999e37d0dab54ca68c0fa66f74582109 [id=e37fd7e36e604a60832acf303f0ecb7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.327147 9506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6323e1c10e7643f8a28f04e285f620e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.334134 9511 tablet_service.cc:1511] Processing CreateTablet for tablet 2c09e9076214405eaa8846fe08158bb3 (DEFAULT_TABLE table=default.Table_6e41cb79b95848ba8c7b1f05f961c7f3 [id=b91fe7c5c3844c109756842f93098699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.335268 9511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c09e9076214405eaa8846fe08158bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.339718 10391 log.cc:1199] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/bd8cd1caaeb249e3bafee7f1f0d4df2f
I20260504 13:56:31.341116 10681 raft_consensus.cc:493] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:31.341563 10681 raft_consensus.cc:515] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.344345 10681 leader_election.cc:290] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:31.345242 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f6b3e969760442283c3191e062c1839" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:31.346488 9618 leader_election.cc:343] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:31.352766 9932 tablet_bootstrap.cc:492] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:31.352751 10890 raft_consensus.cc:515] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.353161 9932 ts_tablet_manager.cc:1403] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.587s user 0.009s sys 0.000s
W20260504 13:56:31.357493 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 44607.9us
I20260504 13:56:31.367280 9799 tablet_service.cc:1558] Processing DeleteTablet for tablet de639fda874248f0853e74ee60aaf758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:28 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:31.374395 9342 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 46fdf0ffaf454689a705949bd39deb26
I20260504 13:56:31.381065 9824 raft_consensus.cc:3060] T 8a985338790e4887af38c49ba013417f P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:31.381114 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f6b3e969760442283c3191e062c1839" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:31.382943 9620 leader_election.cc:343] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:31.383414 9620 leader_election.cc:304] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:31.383688 9828 raft_consensus.cc:3060] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:31.384027 10901 raft_consensus.cc:2749] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:31.368667 10391 ts_tablet_manager.cc:1950] T bd8cd1caaeb249e3bafee7f1f0d4df2f P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:31.398434 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 226) took 3105 ms. Trace:
I20260504 13:56:31.398808 10391 rpcz_store.cc:276] 0504 13:56:28.292850 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.292963 (+ 113us) service_pool.cc:224] Handling call
0504 13:56:30.705204 (+2412241us) ts_tablet_manager.cc:1151] Deleting tablet bd8cd1caaeb249e3bafee7f1f0d4df2f
0504 13:56:30.705237 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:30.889899 (+184662us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:31.046783 (+156884us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:31.398378 (+351595us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2411749}
I20260504 13:56:31.400736 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 227) took 3106 ms. Trace:
I20260504 13:56:31.400960 10890 leader_election.cc:290] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:31.401121 10391 rpcz_store.cc:276] 0504 13:56:28.293933 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.294040 (+ 107us) service_pool.cc:224] Handling call
0504 13:56:31.400059 (+3106019us) ts_tablet_manager.cc:1151] Deleting tablet afad9329cf7745ba8168b2a93f998295
0504 13:56:31.400107 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:31.400260 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3105513}
I20260504 13:56:31.402626 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 228) took 3107 ms. Trace:
I20260504 13:56:31.402971 10391 rpcz_store.cc:276] 0504 13:56:28.295109 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.295238 (+ 129us) service_pool.cc:224] Handling call
0504 13:56:31.402407 (+3107169us) ts_tablet_manager.cc:1151] Deleting tablet 70caac2e256d419dbc2e70cb5dfffe97
0504 13:56:31.402461 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:31.402584 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3106677}
I20260504 13:56:31.412309 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ebc96408d64c2da7dc6c4fc158995c" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:31.413959 9833 raft_consensus.cc:3060] T 184fefa651fc411fb5107cc923c02535 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:31.442495 10922 tablet_replica.cc:333] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:31.443799 10922 raft_consensus.cc:2243] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:31.444504 10922 raft_consensus.cc:2272] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:31.444598 10904 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:31.445185 10904 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:31.451107 10904 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:31.374655 9507 tablet_service.cc:1511] Processing CreateTablet for tablet 7989338bc4bf4bad8fb8fe37587d0a16 (DEFAULT_TABLE table=default.Table_10e92469b9cb4e9b965d0a10cfeecca8 [id=99da5017cead4ddeaa3441064b1ed716]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.458103 9507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7989338bc4bf4bad8fb8fe37587d0a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.464730 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e70772b7ce4f27a941918aaf66a865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.476936 10901 raft_consensus.cc:493] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:31.477358 10901 raft_consensus.cc:515] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.479118 10901 leader_election.cc:290] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:31.480545 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a26c6a2ffb8746a3941a56c7259412e3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:31.481794 9618 leader_election.cc:343] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:31.492080 10918 tablet_replica.cc:333] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:31.492949 10918 raft_consensus.cc:2243] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:31.493737 10918 raft_consensus.cc:2272] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:31.496152 9664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd635a863f7a41d087947f619e41f95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.496982 10918 ts_tablet_manager.cc:1916] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:31.500633 9659 tablet_service.cc:1558] Processing DeleteTablet for tablet 46fdf0ffaf454689a705949bd39deb26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:28 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:31.507551 9342 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35768 (request call id 265) took 4175 ms. Trace:
I20260504 13:56:31.512681 9659 tablet_service.cc:1511] Processing CreateTablet for tablet 6f4c8ffb2175493a981c18676485ac04 (DEFAULT_TABLE table=default.Table_93e03451ea09440d847c57c026230518 [id=6ad5facbe9ca40eeb7746baef5c566a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.513612 9659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f4c8ffb2175493a981c18676485ac04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.507881 9342 rpcz_store.cc:276] 0504 13:56:27.332100 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.448528 (+1116428us) service_pool.cc:224] Handling call
0504 13:56:28.480071 (+ 31543us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b0c46 00007fc27b2c2024 00007fc27b2c1fb2 00007fc27b2c1ee1 00007fc27b2c1e5c 00007fc27b2c1112 00007fc27958bbcc 00007fc27958b006
0504 13:56:28.509717 (+ 29646us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:28.510506 (+ 789us) catalog_manager.cc:2815] Modifying in-memory table state
0504 13:56:28.510613 (+ 107us) catalog_manager.cc:2824] Locking tablets
0504 13:56:28.511191 (+ 578us) catalog_manager.cc:2838] Removing table and tablets from system table
0504 13:56:28.602112 (+ 90921us) catalog_manager.cc:2856] Removing table from by-name map
0504 13:56:28.602384 (+ 272us) catalog_manager.cc:2872] Committing in-memory state
0504 13:56:31.374303 (+2771919us) catalog_manager.cc:2876] Aborting table tasks
0504 13:56:31.507492 (+133189us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:28.556700 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:28.558803 (+ 2103us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:28.558828 (+ 25us) write_op.cc:435] Acquired schema lock
0504 13:56:28.558840 (+ 12us) tablet.cc:662] Decoding operations
0504 13:56:28.559162 (+ 322us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:28.565800 (+ 6638us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:28.565830 (+ 30us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:28.566154 (+ 324us) tablet.cc:701] Row locks acquired
0504 13:56:28.566169 (+ 15us) write_op.cc:260] PREPARE: finished
0504 13:56:28.566291 (+ 122us) write_op.cc:270] Start()
0504 13:56:28.566378 (+ 87us) write_op.cc:276] Timestamp: P: 1777902988566252 usec, L: 0
0504 13:56:28.566402 (+ 24us) op_driver.cc:348] REPLICATION: starting
0504 13:56:28.573593 (+ 7191us) log.cc:844] Serialized 765 byte log entry
0504 13:56:28.588711 (+ 15118us) op_driver.cc:464] REPLICATION: finished
0504 13:56:28.594324 (+ 5613us) write_op.cc:301] APPLY: starting
0504 13:56:28.594377 (+ 53us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:28.594471 (+ 94us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:28.594561 (+ 90us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:28.594823 (+ 262us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:28.595241 (+ 418us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:28.595291 (+ 50us) write_op.cc:312] APPLY: finished
0504 13:56:28.595456 (+ 165us) log.cc:844] Serialized 29 byte log entry
0504 13:56:28.596141 (+ 685us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:28.596210 (+ 69us) write_op.cc:454] Released schema lock
0504 13:56:28.596247 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:28.605447 (+ 9200us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:28.605474 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":31581952,"child_traces":[["op",{"apply.queue_time_us":5479,"apply.run_cpu_time_us":1373,"apply.run_wall_time_us":2115,"num_ops":2,"prepare.queue_time_us":43945,"prepare.run_cpu_time_us":2466,"prepare.run_wall_time_us":48724,"raft.queue_time_us":2560,"raft.run_cpu_time_us":690,"raft.run_wall_time_us":1340,"replication_time_us":22221}]]}
I20260504 13:56:31.578426 9659 tablet_service.cc:1511] Processing CreateTablet for tablet 6677a93d01d54f15a506ae1474bcb501 (DEFAULT_TABLE table=default.Table_4842f50414d34bdb8a10802bc059b421 [id=7fc23620ef7342eea52bcce264eb12b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.579794 9659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6677a93d01d54f15a506ae1474bcb501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.594871 10918 ts_tablet_manager.cc:1929] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:31.602191 10918 log.cc:1199] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/de639fda874248f0853e74ee60aaf758
I20260504 13:56:31.605693 10918 ts_tablet_manager.cc:1950] T de639fda874248f0853e74ee60aaf758 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:31.609990 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet de639fda874248f0853e74ee60aaf758 (table default.Table_0d4595642a604b8f8f57d3eadbbacacb [id=666099fdb1a94091ac6fd99cf58f18e2]) successfully deleted
I20260504 13:56:31.618505 10918 tablet_replica.cc:333] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
W20260504 13:56:31.620780 9338 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet 274bb204041c4e1085395c7bd938aaf1 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5], current schema version=4): Not found: failed to reset TS proxy: No leader found: Tablet 274bb204041c4e1085395c7bd938aaf1 consensus state has no leader
I20260504 13:56:31.622444 10918 raft_consensus.cc:2243] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:31.624344 10918 raft_consensus.cc:2272] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:31.634930 9335 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:38691 (request call id 43) took 3454 ms (client timeout 1000 ms). Trace:
I20260504 13:56:31.636226 10901 raft_consensus.cc:493] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:31.636641 10901 raft_consensus.cc:515] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:31.635304 9335 rpcz_store.cc:269] 0504 13:56:28.180693 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.187155 (+ 6462us) service_pool.cc:224] Handling call
0504 13:56:28.455904 (+268749us) spinlock_profiling.cc:243] Waited 85.4 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:31.634857 (+3178953us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:30.089633 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:30.090158 (+ 525us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:30.090185 (+ 27us) write_op.cc:435] Acquired schema lock
0504 13:56:30.090197 (+ 12us) tablet.cc:662] Decoding operations
0504 13:56:30.090576 (+ 379us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:30.090634 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:30.090668 (+ 34us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:30.090992 (+ 324us) tablet.cc:701] Row locks acquired
0504 13:56:30.091007 (+ 15us) write_op.cc:260] PREPARE: finished
0504 13:56:30.091115 (+ 108us) write_op.cc:270] Start()
0504 13:56:30.091224 (+ 109us) write_op.cc:276] Timestamp: P: 1777902990091074 usec, L: 0
0504 13:56:30.091241 (+ 17us) op_driver.cc:348] REPLICATION: starting
0504 13:56:30.091806 (+ 565us) log.cc:844] Serialized 913 byte log entry
0504 13:56:30.099032 (+ 7226us) op_driver.cc:464] REPLICATION: finished
0504 13:56:30.099311 (+ 279us) write_op.cc:301] APPLY: starting
0504 13:56:30.099362 (+ 51us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:30.099458 (+ 96us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:30.099545 (+ 87us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:30.099806 (+ 261us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:30.099947 (+ 141us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:30.099990 (+ 43us) write_op.cc:312] APPLY: finished
0504 13:56:30.100171 (+ 181us) log.cc:844] Serialized 29 byte log entry
0504 13:56:30.100311 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:30.100373 (+ 62us) write_op.cc:454] Released schema lock
0504 13:56:30.100400 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:30.100610 (+ 210us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:30.100651 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":19899,"reported_tablets":32,"spinlock_wait_cycles":196412672,"child_traces":[["op",{"apply.queue_time_us":151,"apply.run_cpu_time_us":1263,"apply.run_wall_time_us":1264,"num_ops":2,"prepare.queue_time_us":110,"prepare.run_cpu_time_us":2490,"prepare.run_wall_time_us":2490,"raft.queue_time_us":79,"raft.run_cpu_time_us":1076,"raft.run_wall_time_us":1076,"replication_time_us":7707,"spinlock_wait_cycles":2304}]]}
I20260504 13:56:31.638437 10901 leader_election.cc:290] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:31.639196 9335 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_c15418510612433e86cd370f162b6c31" } new_table_name: "default.Table_42f086bc327c42ccad96925c681a386a" modify_external_catalogs: true
I20260504 13:56:31.641389 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a1a77ad56634e068401a2bfbc977761" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:31.643272 9618 leader_election.cc:343] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:31.644696 10918 ts_tablet_manager.cc:1916] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260504 13:56:31.646044 9339 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 38) took 3459 ms (client timeout 1000 ms). Trace:
W20260504 13:56:31.647234 9340 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:55533 (request call id 45) took 3000 ms (client timeout 1000 ms). Trace:
I20260504 13:56:31.650161 9664 tablet_service.cc:1558] Processing DeleteTablet for tablet de639fda874248f0853e74ee60aaf758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:28 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:31.650463 9657 tablet_service.cc:1558] Processing DeleteTablet for tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 with delete_type TABLET_DATA_DELETED (replaced by tablet eef975650610423c88ce8e4cfb0b013b) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:31.665560 9657 tablet_service.cc:1558] Processing DeleteTablet for tablet 8e4e583afc804212bece2f727411cc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:23 UTC) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:31.646716 9339 rpcz_store.cc:269] 0504 13:56:28.186813 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.195665 (+ 8852us) service_pool.cc:224] Handling call
0504 13:56:31.645979 (+3450314us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:29.894792 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:29.895294 (+ 502us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:29.895326 (+ 32us) write_op.cc:435] Acquired schema lock
0504 13:56:29.895340 (+ 14us) tablet.cc:662] Decoding operations
0504 13:56:29.895636 (+ 296us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:29.895685 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:29.895709 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:29.895869 (+ 160us) tablet.cc:701] Row locks acquired
0504 13:56:29.895882 (+ 13us) write_op.cc:260] PREPARE: finished
0504 13:56:29.896013 (+ 131us) write_op.cc:270] Start()
0504 13:56:29.896107 (+ 94us) write_op.cc:276] Timestamp: P: 1777902989895946 usec, L: 0
0504 13:56:29.896126 (+ 19us) op_driver.cc:348] REPLICATION: starting
0504 13:56:29.902203 (+ 6077us) log.cc:844] Serialized 533 byte log entry
0504 13:56:29.912130 (+ 9927us) op_driver.cc:464] REPLICATION: finished
0504 13:56:29.912423 (+ 293us) write_op.cc:301] APPLY: starting
0504 13:56:29.912484 (+ 61us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:29.912564 (+ 80us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:29.912650 (+ 86us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:29.912826 (+ 176us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:29.912948 (+ 122us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:29.913102 (+ 154us) write_op.cc:312] APPLY: finished
0504 13:56:29.913289 (+ 187us) log.cc:844] Serialized 23 byte log entry
0504 13:56:29.913424 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:29.913484 (+ 60us) write_op.cc:454] Released schema lock
0504 13:56:29.913513 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:29.913745 (+ 232us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:29.913770 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":50661,"reported_tablets":36,"child_traces":[["op",{"apply.queue_time_us":153,"apply.run_cpu_time_us":1281,"apply.run_wall_time_us":1270,"num_ops":1,"prepare.queue_time_us":221530,"prepare.run_cpu_time_us":2200,"prepare.run_wall_time_us":15574,"raft.queue_time_us":1312,"raft.run_cpu_time_us":552,"raft.run_wall_time_us":552,"replication_time_us":15792}]]}
I20260504 13:56:31.667301 9339 master_service.cc:954] ReplaceTablet: received request to replace tablet 8bb77106b48a45a2aaa938dc474fa8bd from {username='slave'} at 127.0.0.1:35768
W20260504 13:56:31.668844 9337 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 39) took 2380 ms (client timeout 1000 ms). Trace:
W20260504 13:56:31.670080 9337 rpcz_store.cc:269] 0504 13:56:29.288276 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:29.340629 (+ 52353us) service_pool.cc:224] Handling call
0504 13:56:29.357346 (+ 16717us) spinlock_profiling.cc:243] Waited 8.49 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:31.668666 (+2311320us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68014,"reported_tablets":44,"spinlock_wait_cycles":19534080}
W20260504 13:56:31.673213 9338 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:38691 (request call id 44) took 2109 ms (client timeout 1000 ms). Trace:
I20260504 13:56:31.675315 9337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_1c9b2e63142440a5a45018d187a6cded"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:31.683369 9337 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c9b2e63142440a5a45018d187a6cded in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:31.670174 9340 rpcz_store.cc:269] 0504 13:56:28.647288 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.647921 (+ 633us) service_pool.cc:224] Handling call
0504 13:56:30.874268 (+2226347us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:30.874293 (+ 25us) write_op.cc:454] Released schema lock
0504 13:56:31.647155 (+772862us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:30.855184 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:30.855659 (+ 475us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:30.855686 (+ 27us) write_op.cc:435] Acquired schema lock
0504 13:56:30.855702 (+ 16us) tablet.cc:662] Decoding operations
0504 13:56:30.856077 (+ 375us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:30.856122 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:30.856150 (+ 28us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:30.856454 (+ 304us) tablet.cc:701] Row locks acquired
0504 13:56:30.856470 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:30.856595 (+ 125us) write_op.cc:270] Start()
0504 13:56:30.856682 (+ 87us) write_op.cc:276] Timestamp: P: 1777902990856530 usec, L: 0
0504 13:56:30.856702 (+ 20us) op_driver.cc:348] REPLICATION: starting
0504 13:56:30.857220 (+ 518us) log.cc:844] Serialized 913 byte log entry
0504 13:56:30.863991 (+ 6771us) op_driver.cc:464] REPLICATION: finished
0504 13:56:30.864669 (+ 678us) write_op.cc:301] APPLY: starting
0504 13:56:30.864727 (+ 58us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:30.864846 (+ 119us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:30.864947 (+ 101us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:30.865286 (+ 339us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:30.865458 (+ 172us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:30.865518 (+ 60us) write_op.cc:312] APPLY: finished
0504 13:56:30.865726 (+ 208us) log.cc:844] Serialized 29 byte log entry
0504 13:56:30.865877 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:30.865937 (+ 60us) write_op.cc:454] Released schema lock
0504 13:56:30.865970 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":85939,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":516,"apply.run_cpu_time_us":1488,"apply.run_wall_time_us":1489,"num_ops":2,"prepare.queue_time_us":4753,"prepare.run_cpu_time_us":2323,"prepare.run_wall_time_us":2324,"raft.queue_time_us":1107,"raft.run_cpu_time_us":501,"raft.run_wall_time_us":501,"replication_time_us":7185}]]}
I20260504 13:56:31.699108 10918 ts_tablet_manager.cc:1929] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:31.404702 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 231) took 3055 ms. Trace:
I20260504 13:56:31.699829 10391 rpcz_store.cc:276] 0504 13:56:28.349335 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:28.349449 (+ 114us) service_pool.cc:224] Handling call
0504 13:56:31.404491 (+3055042us) ts_tablet_manager.cc:1151] Deleting tablet c34630d55f4a4e19b33ddf4c13995a5b
0504 13:56:31.404527 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:31.404655 (+ 128us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3054398}
I20260504 13:56:31.701089 10391 tablet_replica.cc:333] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:31.706489 10918 log.cc:1199] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/46fdf0ffaf454689a705949bd39deb26
I20260504 13:56:31.710817 10918 ts_tablet_manager.cc:1950] T 46fdf0ffaf454689a705949bd39deb26 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:31.483280 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a26c6a2ffb8746a3941a56c7259412e3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:31.524121 9806 tablet_service.cc:1511] Processing CreateTablet for tablet 6f4c8ffb2175493a981c18676485ac04 (DEFAULT_TABLE table=default.Table_93e03451ea09440d847c57c026230518 [id=6ad5facbe9ca40eeb7746baef5c566a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.712713 9806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f4c8ffb2175493a981c18676485ac04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.713294 10922 ts_tablet_manager.cc:1916] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:31.531311 10926 raft_consensus.cc:493] T 184fefa651fc411fb5107cc923c02535 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:31.550280 10935 raft_consensus.cc:493] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:31.715451 10926 raft_consensus.cc:515] T 184fefa651fc411fb5107cc923c02535 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.715518 10935 raft_consensus.cc:515] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:31.717335 10926 leader_election.cc:290] T 184fefa651fc411fb5107cc923c02535 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:31.717428 10935 leader_election.cc:290] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:31.719323 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "184fefa651fc411fb5107cc923c02535" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:31.720033 10933 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260504 13:56:31.721153 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274bb204041c4e1085395c7bd938aaf1" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
W20260504 13:56:31.722093 9620 leader_election.cc:343] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:31.722491 9620 leader_election.cc:304] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:31.723433 10901 raft_consensus.cc:2749] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:31.614424 9799 tablet_service.cc:1511] Processing CreateTablet for tablet 6677a93d01d54f15a506ae1474bcb501 (DEFAULT_TABLE table=default.Table_4842f50414d34bdb8a10802bc059b421 [id=7fc23620ef7342eea52bcce264eb12b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:31.629513 9803 tablet_service.cc:1558] Processing DeleteTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:16 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:31.629968 9796 tablet_service.cc:1558] Processing DeleteTablet for tablet 50e129980cd84f6ca22cbf2addb06230 with delete_type TABLET_DATA_DELETED (replaced by tablet c47b6e3b4d0c45c28384f7aae6b749de) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:31.632241 9800 tablet_service.cc:1558] Processing DeleteTablet for tablet c34630d55f4a4e19b33ddf4c13995a5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:19 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:31.639977 9805 tablet_service.cc:1558] Processing DeleteTablet for tablet 604f1ba5f71041a7877daff8972a994a with delete_type TABLET_DATA_DELETED (replaced by tablet 2ddf890c17964e7f90409ebe3c4034ef) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:31.640658 9808 tablet_service.cc:1558] Processing DeleteTablet for tablet afad9329cf7745ba8168b2a93f998295 with delete_type TABLET_DATA_DELETED (replaced by tablet af5c5a6cae4e4b6ea3eece2e21a587ec) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:31.640771 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a1a77ad56634e068401a2bfbc977761" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:31.641886 9798 tablet_service.cc:1558] Processing DeleteTablet for tablet 030b35c3409143108c34ab1f0ee1b96d with delete_type TABLET_DATA_DELETED (replaced by tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:31.642051 9802 tablet_service.cc:1558] Processing DeleteTablet for tablet 930078a10afa4dd987ad021cf9609514 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:31.642163 9812 tablet_service.cc:1558] Processing DeleteTablet for tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:24 UTC) from {username='slave'} at 127.0.0.1:56380
W20260504 13:56:31.702822 9666 consensus_meta.cc:287] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6: Time spent flushing consensus metadata: real 0.603s user 0.005s sys 0.000s
W20260504 13:56:31.729127 9620 leader_election.cc:343] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:31.730427 10918 tablet_replica.cc:333] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:31.731086 9199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:31.729476 9620 leader_election.cc:304] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:31.734251 10901 raft_consensus.cc:2749] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:31.736770 10918 raft_consensus.cc:2243] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:31.737335 10918 raft_consensus.cc:2272] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:31.742334 10918 ts_tablet_manager.cc:1916] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:31.753793 10013 raft_consensus.cc:359] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.754529 10013 raft_consensus.cc:385] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:31.754880 10013 raft_consensus.cc:740] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:31.661813 9801 tablet_service.cc:1558] Processing DeleteTablet for tablet 46fdf0ffaf454689a705949bd39deb26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:28 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:31.668483 9879 heartbeater.cc:344] Connected to a master server at 127.8.238.252:44781
I20260504 13:56:31.764865 10788 raft_consensus.cc:493] T 8a985338790e4887af38c49ba013417f P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7b79fe7dd8934dd996393a2d96eabf00)
I20260504 13:56:31.765475 10788 raft_consensus.cc:515] T 8a985338790e4887af38c49ba013417f P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.772253 10788 leader_election.cc:290] T 8a985338790e4887af38c49ba013417f P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:31.773310 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a985338790e4887af38c49ba013417f" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:31.780622 10918 ts_tablet_manager.cc:1929] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:31.783885 10918 log.cc:1199] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/8e4e583afc804212bece2f727411cc14
I20260504 13:56:31.785032 10918 ts_tablet_manager.cc:1950] T 8e4e583afc804212bece2f727411cc14 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:31.815904 9931 tablet_bootstrap.cc:700] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888: Time spent opening tablet: real 0.421s user 0.000s sys 0.003s
I20260504 13:56:31.816499 9931 tablet_bootstrap.cc:654] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:31.818890 10940 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:31.819242 9932 raft_consensus.cc:359] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.819941 9932 raft_consensus.cc:385] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:31.819514 10940 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:31.820199 9932 raft_consensus.cc:740] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:31.820945 9932 consensus_queue.cc:260] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.822980 9932 ts_tablet_manager.cc:1434] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.470s user 0.005s sys 0.000s
I20260504 13:56:31.825274 10777 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "5beceb6317444247b31b44a6581707c6" instance_seqno: 1777902965763794) as {username='slave'} at 127.8.238.194:56157; Asking this server to re-register.
I20260504 13:56:31.828905 9732 heartbeater.cc:461] Registering TS with master...
I20260504 13:56:31.830027 10940 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:31.836735 9340 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_d6a5a22d78264e799e80d31d5749600a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:31.839419 9340 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6a5a22d78264e799e80d31d5749600a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:31.762414 9338 rpcz_store.cc:269] 0504 13:56:29.563669 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:29.754785 (+191116us) service_pool.cc:224] Handling call
0504 13:56:29.787998 (+ 33213us) spinlock_profiling.cc:243] Waited 28.6 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:31.120453 (+1332455us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:31.120478 (+ 25us) write_op.cc:454] Released schema lock
0504 13:56:31.673079 (+552601us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:31.102209 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:31.102655 (+ 446us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:31.102680 (+ 25us) write_op.cc:435] Acquired schema lock
0504 13:56:31.102694 (+ 14us) tablet.cc:662] Decoding operations
0504 13:56:31.103163 (+ 469us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:31.103245 (+ 82us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:31.103270 (+ 25us) tablet.cc:685] Acquiring locks for 3 operations
0504 13:56:31.103557 (+ 287us) tablet.cc:701] Row locks acquired
0504 13:56:31.103576 (+ 19us) write_op.cc:260] PREPARE: finished
0504 13:56:31.103688 (+ 112us) write_op.cc:270] Start()
0504 13:56:31.103781 (+ 93us) write_op.cc:276] Timestamp: P: 1777902991103645 usec, L: 0
0504 13:56:31.103800 (+ 19us) op_driver.cc:348] REPLICATION: starting
0504 13:56:31.104329 (+ 529us) log.cc:844] Serialized 1293 byte log entry
0504 13:56:31.113703 (+ 9374us) op_driver.cc:464] REPLICATION: finished
0504 13:56:31.114057 (+ 354us) write_op.cc:301] APPLY: starting
0504 13:56:31.114117 (+ 60us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:31.114260 (+ 143us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:31.114352 (+ 92us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:31.114696 (+ 344us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:31.114853 (+ 157us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0504 13:56:31.114904 (+ 51us) write_op.cc:312] APPLY: finished
0504 13:56:31.115123 (+ 219us) log.cc:844] Serialized 35 byte log entry
0504 13:56:31.115274 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:31.115348 (+ 74us) write_op.cc:454] Released schema lock
0504 13:56:31.115377 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":30271,"reported_tablets":34,"spinlock_wait_cycles":65819520,"child_traces":[["op",{"apply.queue_time_us":203,"apply.run_cpu_time_us":1484,"apply.run_wall_time_us":1483,"num_ops":3,"prepare.queue_time_us":85,"prepare.run_cpu_time_us":2386,"prepare.run_wall_time_us":2386,"raft.queue_time_us":1155,"raft.run_cpu_time_us":614,"raft.run_wall_time_us":614,"replication_time_us":9808}]]}
I20260504 13:56:31.857851 10013 consensus_queue.cc:260] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.859207 9799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6677a93d01d54f15a506ae1474bcb501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:31.871905 9932 tablet_bootstrap.cc:492] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:31.877383 9932 tablet_bootstrap.cc:654] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:31.896304 9932 tablet_bootstrap.cc:492] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:31.896406 10922 ts_tablet_manager.cc:1929] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:31.896763 10922 log.cc:1199] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/de639fda874248f0853e74ee60aaf758
I20260504 13:56:31.896837 9932 ts_tablet_manager.cc:1403] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.007s
I20260504 13:56:31.897578 10922 ts_tablet_manager.cc:1950] T de639fda874248f0853e74ee60aaf758 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:31.899319 9932 raft_consensus.cc:359] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.900790 9932 raft_consensus.cc:385] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:31.901326 9932 raft_consensus.cc:740] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:31.902377 9932 consensus_queue.cc:260] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.903354 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet de639fda874248f0853e74ee60aaf758 (table default.Table_0d4595642a604b8f8f57d3eadbbacacb [id=666099fdb1a94091ac6fd99cf58f18e2]) successfully deleted
W20260504 13:56:31.903982 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f because the tablet was not found. No further retry: Not found: Tablet not found: bd8cd1caaeb249e3bafee7f1f0d4df2f
W20260504 13:56:31.904533 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 50e129980cd84f6ca22cbf2addb06230 because the tablet was not found. No further retry: Not found: Tablet not found: 50e129980cd84f6ca22cbf2addb06230
I20260504 13:56:31.904865 10922 tablet_replica.cc:333] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
W20260504 13:56:31.905140 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet c34630d55f4a4e19b33ddf4c13995a5b because the tablet was not found. No further retry: Not found: Tablet not found: c34630d55f4a4e19b33ddf4c13995a5b
I20260504 13:56:31.905553 9932 ts_tablet_manager.cc:1434] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
W20260504 13:56:31.905654 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet afad9329cf7745ba8168b2a93f998295 because the tablet was not found. No further retry: Not found: Tablet not found: afad9329cf7745ba8168b2a93f998295
I20260504 13:56:31.905718 10922 raft_consensus.cc:2243] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:31.906152 10922 raft_consensus.cc:2272] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:31.906178 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 604f1ba5f71041a7877daff8972a994a because the tablet was not found. No further retry: Not found: Tablet not found: 604f1ba5f71041a7877daff8972a994a
I20260504 13:56:31.906327 9932 tablet_bootstrap.cc:492] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:31.906713 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 030b35c3409143108c34ab1f0ee1b96d because the tablet was not found. No further retry: Not found: Tablet not found: 030b35c3409143108c34ab1f0ee1b96d
W20260504 13:56:31.907704 9330 catalog_manager.cc:4969] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 930078a10afa4dd987ad021cf9609514 because the tablet was not found. No further retry: Not found: Tablet not found: 930078a10afa4dd987ad021cf9609514
I20260504 13:56:31.908015 10922 ts_tablet_manager.cc:1916] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:31.916427 9932 tablet_bootstrap.cc:654] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:31.920265 9931 tablet_bootstrap.cc:492] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:31.920639 9931 ts_tablet_manager.cc:1403] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.629s user 0.014s sys 0.025s
I20260504 13:56:31.922801 9931 raft_consensus.cc:359] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.923384 9931 raft_consensus.cc:385] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:31.923624 9931 raft_consensus.cc:740] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:31.923741 10922 ts_tablet_manager.cc:1929] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:31.924111 10922 log.cc:1199] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/a1aba9a2a58b4e6b9e87f8ec80fcb6f1
I20260504 13:56:31.924754 10922 ts_tablet_manager.cc:1950] T a1aba9a2a58b4e6b9e87f8ec80fcb6f1 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:31.924362 9931 consensus_queue.cc:260] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.929260 10681 raft_consensus.cc:493] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:31.929792 10922 tablet_replica.cc:333] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:31.929678 10681 raft_consensus.cc:515] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.930603 10922 raft_consensus.cc:2243] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:31.931061 10922 raft_consensus.cc:2272] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:31.931454 10681 leader_election.cc:290] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:31.931829 9931 ts_tablet_manager.cc:1434] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260504 13:56:31.932684 9931 tablet_bootstrap.cc:492] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:31.933032 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47b6e3b4d0c45c28384f7aae6b749de" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:31.934365 9620 leader_election.cc:343] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:31.937273 9932 tablet_bootstrap.cc:492] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:31.937659 9932 ts_tablet_manager.cc:1403] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.000s
I20260504 13:56:31.938634 10776 ts_manager.cc:194] Registered new tserver with Master: 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:31.939968 9932 raft_consensus.cc:359] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.940547 9932 raft_consensus.cc:385] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:31.940783 9932 raft_consensus.cc:740] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:31.941306 9932 consensus_queue.cc:260] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.943470 9932 ts_tablet_manager.cc:1434] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260504 13:56:31.944320 9932 tablet_bootstrap.cc:492] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:31.949443 9932 tablet_bootstrap.cc:654] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:31.705385 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet bd8cd1caaeb249e3bafee7f1f0d4df2f (table default.Table_47c9ed0b601146b687a576ec89c8b626 [id=b75b9e5edced4443944af4087b77e54d]) successfully deleted
I20260504 13:56:31.714243 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 46fdf0ffaf454689a705949bd39deb26 (table default.Table_276161949f364064b8702a62c6beb9e4 [id=e68beb5025184dcb9c24037b6b595293]) successfully deleted
W20260504 13:56:31.951406 9328 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 246027us
W20260504 13:56:31.951979 9329 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 237739us
W20260504 13:56:31.952344 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet afad9329cf7745ba8168b2a93f998295 because the tablet was not found. No further retry: Not found: Tablet not found: afad9329cf7745ba8168b2a93f998295
W20260504 13:56:31.952877 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 70caac2e256d419dbc2e70cb5dfffe97 because the tablet was not found. No further retry: Not found: Tablet not found: 70caac2e256d419dbc2e70cb5dfffe97
W20260504 13:56:31.953286 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet de639fda874248f0853e74ee60aaf758 because the tablet was not found. No further retry: Not found: Tablet not found: de639fda874248f0853e74ee60aaf758
W20260504 13:56:31.953784 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 because the tablet was not found. No further retry: Not found: Tablet not found: 41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:31.954447 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 8e4e583afc804212bece2f727411cc14 (table default.Table_a8bc83249ad949c49e759a07818e2c02 [id=4182fad7cb2344d88498c26192164a10]) successfully deleted
I20260504 13:56:31.955257 10922 ts_tablet_manager.cc:1916] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:31.955785 9661 tablet_service.cc:1558] Processing DeleteTablet for tablet 7c9030802900499a90c4dbaf5c6936a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6dd139c347f7406aac340cc6ac994273) from {username='slave'} at 127.0.0.1:40388
W20260504 13:56:31.761581 9336 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:55533 (request call id 46) took 2123 ms (client timeout 1000 ms). Trace:
W20260504 13:56:31.830874 9344 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 37) took 4664 ms (client timeout 1000 ms). Trace:
I20260504 13:56:31.957072 9931 tablet_bootstrap.cc:654] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:31.956454 9344 rpcz_store.cc:269] 0504 13:56:27.166536 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:27.912890 (+746354us) service_pool.cc:224] Handling call
0504 13:56:31.830809 (+3917919us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:29.363720 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:29.364244 (+ 524us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:29.364276 (+ 32us) write_op.cc:435] Acquired schema lock
0504 13:56:29.364293 (+ 17us) tablet.cc:662] Decoding operations
0504 13:56:29.364623 (+ 330us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:29.364690 (+ 67us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:29.364725 (+ 35us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:29.364909 (+ 184us) tablet.cc:701] Row locks acquired
0504 13:56:29.364925 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:29.365059 (+ 134us) write_op.cc:270] Start()
0504 13:56:29.365164 (+ 105us) write_op.cc:276] Timestamp: P: 1777902989364987 usec, L: 0
0504 13:56:29.365187 (+ 23us) op_driver.cc:348] REPLICATION: starting
0504 13:56:29.365797 (+ 610us) log.cc:844] Serialized 533 byte log entry
0504 13:56:29.425726 (+ 59929us) op_driver.cc:464] REPLICATION: finished
0504 13:56:29.426081 (+ 355us) write_op.cc:301] APPLY: starting
0504 13:56:29.426147 (+ 66us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:29.426234 (+ 87us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:29.426322 (+ 88us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:29.426511 (+ 189us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:29.426662 (+ 151us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:29.426744 (+ 82us) write_op.cc:312] APPLY: finished
0504 13:56:29.426928 (+ 184us) log.cc:844] Serialized 23 byte log entry
0504 13:56:29.427090 (+ 162us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:29.427163 (+ 73us) write_op.cc:454] Released schema lock
0504 13:56:29.427213 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:29.451224 (+ 24011us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:29.451254 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6803,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":204,"apply.run_cpu_time_us":1307,"apply.run_wall_time_us":1308,"num_ops":1,"prepare.queue_time_us":140631,"prepare.run_cpu_time_us":2484,"prepare.run_wall_time_us":87469,"raft.queue_time_us":37500,"raft.run_cpu_time_us":634,"raft.run_wall_time_us":1360,"replication_time_us":60423}]]}
W20260504 13:56:31.959641 9336 rpcz_store.cc:269] 0504 13:56:29.637819 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:29.819525 (+181706us) service_pool.cc:224] Handling call
0504 13:56:29.888141 (+ 68616us) spinlock_profiling.cc:243] Waited 64.5 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:31.761509 (+1873368us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3788,"reported_tablets":30,"spinlock_wait_cycles":148370176}
I20260504 13:56:31.928138 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1 (table default.Table_7f9a1296bc424f7aa21625c670da0dd0 [id=a62d0dfdbe7741248a4d5d6614c712eb]) successfully deleted
W20260504 13:56:31.968595 9329 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 7c9030802900499a90c4dbaf5c6936a4 because the tablet was not found. No further retry: Not found: Tablet not found: 7c9030802900499a90c4dbaf5c6936a4
I20260504 13:56:31.973559 10777 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" instance_seqno: 1777902968507939) as {username='slave'} at 127.8.238.195:50343; Asking this server to re-register.
I20260504 13:56:31.975874 9932 tablet_bootstrap.cc:492] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:31.976266 9932 ts_tablet_manager.cc:1403] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.002s
W20260504 13:56:31.980558 9328 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet a862a43c5ac443d9bac98e0472ca23b2 consensus state has no leader
I20260504 13:56:31.981654 9932 raft_consensus.cc:359] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.982254 9932 raft_consensus.cc:385] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:31.982503 9932 raft_consensus.cc:740] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:31.985761 9932 consensus_queue.cc:260] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:31.990386 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 341 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:31.991148 9344 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_24398dd8ee63413ebb585587353ea155"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:31.992975 9344 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24398dd8ee63413ebb585587353ea155 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:31.998284 9932 ts_tablet_manager.cc:1434] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20260504 13:56:31.999089 9932 tablet_bootstrap.cc:492] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:32.004348 9932 tablet_bootstrap.cc:654] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:32.011646 9336 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35768:
name: "default.Table_dd11ea89ad7248c39ec114af58cf598f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:32.012812 9931 tablet_bootstrap.cc:492] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:32.013192 9931 ts_tablet_manager.cc:1403] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.081s user 0.017s sys 0.006s
W20260504 13:56:32.013242 9336 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd11ea89ad7248c39ec114af58cf598f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:32.015363 9931 raft_consensus.cc:359] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.015980 9931 raft_consensus.cc:385] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:32.016225 9931 raft_consensus.cc:740] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:32.016897 9931 consensus_queue.cc:260] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.017786 10922 ts_tablet_manager.cc:1929] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:32.018184 10922 log.cc:1199] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/46fdf0ffaf454689a705949bd39deb26
I20260504 13:56:32.018936 10922 ts_tablet_manager.cc:1950] T 46fdf0ffaf454689a705949bd39deb26 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:32.019272 9931 ts_tablet_manager.cc:1434] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260504 13:56:32.020226 9931 tablet_bootstrap.cc:492] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:32.025005 9931 tablet_bootstrap.cc:654] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:32.041033 9339 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 8bb77106b48a45a2aaa938dc474fa8bd
I20260504 13:56:32.041136 9337 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35768 (request call id 281) took 2304 ms. Trace:
I20260504 13:56:32.041529 9330 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 46fdf0ffaf454689a705949bd39deb26 (table default.Table_276161949f364064b8702a62c6beb9e4 [id=e68beb5025184dcb9c24037b6b595293]) successfully deleted
I20260504 13:56:32.042243 9339 catalog_manager.cc:6741] ReplaceTablet: tablet 8bb77106b48a45a2aaa938dc474fa8bd deleted and replaced by tablet f337e9698fe64282a684e90d9c61da5a
I20260504 13:56:32.042562 9339 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:35768 (request call id 280) took 2409 ms. Trace:
I20260504 13:56:32.043252 9795 tablet_service.cc:1558] Processing DeleteTablet for tablet 8bb77106b48a45a2aaa938dc474fa8bd with delete_type TABLET_DATA_DELETED (replaced by tablet f337e9698fe64282a684e90d9c61da5a) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:32.044276 9661 tablet_service.cc:1558] Processing DeleteTablet for tablet 8bb77106b48a45a2aaa938dc474fa8bd with delete_type TABLET_DATA_DELETED (replaced by tablet f337e9698fe64282a684e90d9c61da5a) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:32.041415 9337 rpcz_store.cc:276] 0504 13:56:29.737212 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:31.674085 (+1936873us) service_pool.cc:224] Handling call
0504 13:56:31.775115 (+101030us) catalog_manager.cc:2101] Acquired catalog manager lock
0504 13:56:31.781659 (+ 6544us) catalog_manager.cc:2147] Created new table and tablet info
0504 13:56:32.040570 (+258911us) catalog_manager.cc:2207] Wrote table and tablets to system table
0504 13:56:32.040839 (+ 269us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0504 13:56:32.041085 (+ 246us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:31.782924 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:31.783418 (+ 494us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:31.783448 (+ 30us) write_op.cc:435] Acquired schema lock
0504 13:56:31.783464 (+ 16us) tablet.cc:662] Decoding operations
0504 13:56:31.783789 (+ 325us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:31.783846 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:31.783873 (+ 27us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:31.784106 (+ 233us) tablet.cc:701] Row locks acquired
0504 13:56:31.784126 (+ 20us) write_op.cc:260] PREPARE: finished
0504 13:56:31.784241 (+ 115us) write_op.cc:270] Start()
0504 13:56:31.784331 (+ 90us) write_op.cc:276] Timestamp: P: 1777902991784189 usec, L: 0
0504 13:56:31.784350 (+ 19us) op_driver.cc:348] REPLICATION: starting
0504 13:56:31.784866 (+ 516us) log.cc:844] Serialized 433 byte log entry
0504 13:56:32.035620 (+250754us) op_driver.cc:464] REPLICATION: finished
0504 13:56:32.038999 (+ 3379us) write_op.cc:301] APPLY: starting
0504 13:56:32.039059 (+ 60us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:32.039162 (+ 103us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:32.039258 (+ 96us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:32.039711 (+ 453us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:32.039884 (+ 173us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0504 13:56:32.039943 (+ 59us) write_op.cc:312] APPLY: finished
0504 13:56:32.040131 (+ 188us) log.cc:844] Serialized 29 byte log entry
0504 13:56:32.040282 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:32.040350 (+ 68us) write_op.cc:454] Released schema lock
0504 13:56:32.040391 (+ 41us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3257,"apply.run_cpu_time_us":1570,"apply.run_wall_time_us":1569,"num_ops":2,"prepare.queue_time_us":176,"prepare.run_cpu_time_us":2147,"prepare.run_wall_time_us":3040,"replication_time_us":251188,"threads_started":1}]]}
I20260504 13:56:32.045961 9337 master_service.cc:954] ReplaceTablet: received request to replace tablet 21d6358b451d4b8d87a22e30e0780f44 from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:32.042843 9339 rpcz_store.cc:276] 0504 13:56:29.633088 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:31.667199 (+2034111us) service_pool.cc:224] Handling call
0504 13:56:32.042516 (+375317us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:31.777762 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:31.778264 (+ 502us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:31.778292 (+ 28us) write_op.cc:435] Acquired schema lock
0504 13:56:31.778308 (+ 16us) tablet.cc:662] Decoding operations
0504 13:56:31.778658 (+ 350us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:31.778710 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:31.778744 (+ 34us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:31.778963 (+ 219us) tablet.cc:701] Row locks acquired
0504 13:56:31.778979 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:31.779094 (+ 115us) write_op.cc:270] Start()
0504 13:56:31.779202 (+ 108us) write_op.cc:276] Timestamp: P: 1777902991779041 usec, L: 0
0504 13:56:31.779223 (+ 21us) op_driver.cc:348] REPLICATION: starting
0504 13:56:31.779744 (+ 521us) log.cc:844] Serialized 628 byte log entry
0504 13:56:32.035354 (+255610us) op_driver.cc:464] REPLICATION: finished
0504 13:56:32.036891 (+ 1537us) write_op.cc:301] APPLY: starting
0504 13:56:32.036948 (+ 57us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:32.037052 (+ 104us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:32.037132 (+ 80us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:32.037504 (+ 372us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:32.037749 (+ 245us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:32.037917 (+ 168us) write_op.cc:312] APPLY: finished
0504 13:56:32.038138 (+ 221us) log.cc:844] Serialized 29 byte log entry
0504 13:56:32.038309 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:32.038389 (+ 80us) write_op.cc:454] Released schema lock
0504 13:56:32.038428 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:32.038683 (+ 255us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:32.038716 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1316,"apply.run_cpu_time_us":1684,"apply.run_wall_time_us":1759,"num_ops":2,"prepare.queue_time_us":102,"prepare.run_cpu_time_us":2216,"prepare.run_wall_time_us":2216,"replication_time_us":256039}]]}
I20260504 13:56:32.049292 10918 tablet_replica.cc:333] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:32.050325 10918 raft_consensus.cc:2243] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:32.050871 10918 raft_consensus.cc:2272] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:32.052908 10918 ts_tablet_manager.cc:1916] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260504 13:56:32.069813 9330 catalog_manager.cc:4982] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): delete failed for tablet 8bb77106b48a45a2aaa938dc474fa8bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bb77106b48a45a2aaa938dc474fa8bd already in progress: creating tablet
I20260504 13:56:32.071964 9340 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35768 (request call id 282) took 2311 ms. Trace:
I20260504 13:56:32.072259 9340 rpcz_store.cc:276] 0504 13:56:29.760707 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:31.836377 (+2075670us) service_pool.cc:224] Handling call
0504 13:56:31.849451 (+ 13074us) catalog_manager.cc:2101] Acquired catalog manager lock
0504 13:56:31.850887 (+ 1436us) catalog_manager.cc:2147] Created new table and tablet info
0504 13:56:32.071270 (+220383us) catalog_manager.cc:2207] Wrote table and tablets to system table
0504 13:56:32.071680 (+ 410us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0504 13:56:32.071914 (+ 234us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:31.852081 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:31.852514 (+ 433us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:31.852549 (+ 35us) write_op.cc:435] Acquired schema lock
0504 13:56:31.852564 (+ 15us) tablet.cc:662] Decoding operations
0504 13:56:31.852859 (+ 295us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:31.852898 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:31.852921 (+ 23us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:31.853124 (+ 203us) tablet.cc:701] Row locks acquired
0504 13:56:31.853141 (+ 17us) write_op.cc:260] PREPARE: finished
0504 13:56:31.853252 (+ 111us) write_op.cc:270] Start()
0504 13:56:31.853340 (+ 88us) write_op.cc:276] Timestamp: P: 1777902991853195 usec, L: 0
0504 13:56:31.853362 (+ 22us) op_driver.cc:348] REPLICATION: starting
0504 13:56:31.853885 (+ 523us) log.cc:844] Serialized 433 byte log entry
0504 13:56:32.067779 (+213894us) op_driver.cc:464] REPLICATION: finished
0504 13:56:32.069061 (+ 1282us) write_op.cc:301] APPLY: starting
0504 13:56:32.069110 (+ 49us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:32.069196 (+ 86us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:32.069322 (+ 126us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:32.069746 (+ 424us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:32.069879 (+ 133us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0504 13:56:32.069928 (+ 49us) write_op.cc:312] APPLY: finished
0504 13:56:32.070098 (+ 170us) log.cc:844] Serialized 29 byte log entry
0504 13:56:32.070226 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:32.070280 (+ 54us) write_op.cc:454] Released schema lock
0504 13:56:32.070310 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:32.070516 (+ 206us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:32.070539 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1124,"apply.run_cpu_time_us":1440,"apply.run_wall_time_us":1439,"num_ops":2,"prepare.queue_time_us":83,"prepare.run_cpu_time_us":2002,"prepare.run_wall_time_us":2002,"replication_time_us":214339}]]}
I20260504 13:56:32.076452 9337 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:35768 (request call id 285) took 2029 ms. Trace:
I20260504 13:56:32.076772 9337 rpcz_store.cc:276] 0504 13:56:30.046724 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:32.045848 (+1999124us) service_pool.cc:224] Handling call
0504 13:56:32.070299 (+ 24451us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b3491 00007fc27b2de227 00007fc27b2de1b2 00007fc27b2de0e1 00007fc27b2de05c 00007fc27b2dd312 00007fc27958bbcc 00007fc27958b006
0504 13:56:32.076402 (+ 6103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54127872}
I20260504 13:56:32.087512 9339 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_3e6fece0bfdb447dae80b4598b049417" } new_table_name: "default.Table_9b40ba316993413181a9743fb7bab7fb" modify_external_catalogs: true
I20260504 13:56:32.092398 9932 tablet_bootstrap.cc:492] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:32.092809 9932 ts_tablet_manager.cc:1403] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.094s user 0.004s sys 0.024s
I20260504 13:56:32.095134 9932 raft_consensus.cc:359] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.116231 9932 raft_consensus.cc:385] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260504 13:56:32.116519 9339 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e53381ee0e7f4e9781b833b435bca831 consensus state has no leader
I20260504 13:56:32.116505 9932 raft_consensus.cc:740] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:32.116482 9344 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35768 (request call id 283) took 2154 ms. Trace:
I20260504 13:56:32.117964 9932 consensus_queue.cc:260] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:31.974368 10690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.8.238.254:36145, user_credentials={real_user=slave}} blocked reactor thread for 145863us
I20260504 13:56:32.121083 9932 ts_tablet_manager.cc:1434] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.028s user 0.003s sys 0.003s
I20260504 13:56:32.124856 9473 leader_election.cc:304] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:32.109617 10814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 341 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
W20260504 13:56:32.127465 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet c34630d55f4a4e19b33ddf4c13995a5b because the tablet was not found. No further retry: Not found: Tablet not found: c34630d55f4a4e19b33ddf4c13995a5b
I20260504 13:56:32.127534 10814 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 806407748d564439807fe5d73db9abda in term 1.
I20260504 13:56:32.129878 9173 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda; no voters: ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:32.130764 10940 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 13:56:32.131068 10940 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Starting leader election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:32.131410 9880 heartbeater.cc:461] Registering TS with master...
I20260504 13:56:32.131384 10940 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 1 FOLLOWER]: Advancing to term 2
I20260504 13:56:32.134325 10681 raft_consensus.cc:493] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:32.134928 10681 raft_consensus.cc:515] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.139707 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:32.143280 10681 leader_election.cc:290] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:32.139004 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a629c2a69084b6b98984c8d374eeb1e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:32.145925 9620 leader_election.cc:343] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:32.150380 10940 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:32.157234 9932 tablet_bootstrap.cc:492] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:32.162346 9505 tablet_service.cc:1511] Processing CreateTablet for tablet bd635a863f7a41d087947f619e41f95c (DEFAULT_TABLE table=default.Table_849bbe7902ca44969c5208828774fa06 [id=c146e84e0e844deda9af40562f980047]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:32.164315 9505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd635a863f7a41d087947f619e41f95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:32.166193 9336 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35768 (request call id 284) took 2138 ms. Trace:
I20260504 13:56:32.167567 9506 tablet_service.cc:1558] Processing DeleteTablet for tablet de639fda874248f0853e74ee60aaf758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:28 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:32.168196 9339 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_2d5a582adb634895a55193dd861f3b60" } modify_external_catalogs: true
I20260504 13:56:32.168676 9339 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_2d5a582adb634895a55193dd861f3b60" } modify_external_catalogs: true
I20260504 13:56:32.168768 9504 tablet_service.cc:1558] Processing DeleteTablet for tablet 604f1ba5f71041a7877daff8972a994a with delete_type TABLET_DATA_DELETED (replaced by tablet 2ddf890c17964e7f90409ebe3c4034ef) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:32.171164 9932 tablet_bootstrap.cc:654] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:32.172150 9931 tablet_bootstrap.cc:492] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:32.172636 9340 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_3bc454a7435a4553be5a9bbcfec24eb3" } modify_external_catalogs: true
I20260504 13:56:32.173139 9340 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35768:
table { table_name: "default.Table_3bc454a7435a4553be5a9bbcfec24eb3" } modify_external_catalogs: true
I20260504 13:56:32.176234 9508 tablet_service.cc:1511] Processing CreateTablet for tablet 6f4c8ffb2175493a981c18676485ac04 (DEFAULT_TABLE table=default.Table_93e03451ea09440d847c57c026230518 [id=6ad5facbe9ca40eeb7746baef5c566a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:32.178424 9509 tablet_service.cc:1511] Processing CreateTablet for tablet 6677a93d01d54f15a506ae1474bcb501 (DEFAULT_TABLE table=default.Table_4842f50414d34bdb8a10802bc059b421 [id=7fc23620ef7342eea52bcce264eb12b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:32.179934 9510 tablet_service.cc:1558] Processing DeleteTablet for tablet 46fdf0ffaf454689a705949bd39deb26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:28 UTC) from {username='slave'} at 127.0.0.1:52642
W20260504 13:56:32.181671 9511 consensus_meta.cc:287] T 2c09e9076214405eaa8846fe08158bb3 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent flushing consensus metadata: real 0.833s user 0.007s sys 0.000s
I20260504 13:56:32.184070 9512 tablet_service.cc:1558] Processing DeleteTablet for tablet 930078a10afa4dd987ad021cf9609514 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:32.184412 9519 tablet_service.cc:1558] Processing DeleteTablet for tablet baf07a970a7c4a35b3c849b06d6c88d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:24 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:32.184952 9517 tablet_service.cc:1511] Processing CreateTablet for tablet e0e70772b7ce4f27a941918aaf66a865 (DEFAULT_TABLE table=default.Table_60fe6ee7a2414e2aa8ea2d57ee4a827d [id=1de44a1bbf06492084cfd70cf540b213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:32.116837 9344 rpcz_store.cc:276] 0504 13:56:29.961732 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:31.960501 (+1998769us) service_pool.cc:224] Handling call
0504 13:56:31.991055 (+ 30554us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b0936 00007fc27b2bd524 00007fc27b2bd4b2 00007fc27b2bd3e1 00007fc27b2bd35c 00007fc27b2bc612 00007fc27958bbcc 00007fc27958b006
0504 13:56:31.993429 (+ 2374us) catalog_manager.cc:2101] Acquired catalog manager lock
0504 13:56:32.017250 (+ 23821us) catalog_manager.cc:2147] Created new table and tablet info
0504 13:56:32.099742 (+ 82492us) catalog_manager.cc:2207] Wrote table and tablets to system table
0504 13:56:32.116121 (+ 16379us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0504 13:56:32.116430 (+ 309us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:32.018723 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:32.019197 (+ 474us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:32.019224 (+ 27us) write_op.cc:435] Acquired schema lock
0504 13:56:32.019239 (+ 15us) tablet.cc:662] Decoding operations
0504 13:56:32.019555 (+ 316us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:32.019593 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:32.019618 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:32.019856 (+ 238us) tablet.cc:701] Row locks acquired
0504 13:56:32.019871 (+ 15us) write_op.cc:260] PREPARE: finished
0504 13:56:32.019981 (+ 110us) write_op.cc:270] Start()
0504 13:56:32.020062 (+ 81us) write_op.cc:276] Timestamp: P: 1777902992019914 usec, L: 0
0504 13:56:32.020078 (+ 16us) op_driver.cc:348] REPLICATION: starting
0504 13:56:32.020569 (+ 491us) log.cc:844] Serialized 433 byte log entry
0504 13:56:32.095021 (+ 74452us) op_driver.cc:464] REPLICATION: finished
0504 13:56:32.096647 (+ 1626us) write_op.cc:301] APPLY: starting
0504 13:56:32.096707 (+ 60us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:32.096801 (+ 94us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:32.096895 (+ 94us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:32.097253 (+ 358us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:32.097396 (+ 143us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0504 13:56:32.097439 (+ 43us) write_op.cc:312] APPLY: finished
0504 13:56:32.097591 (+ 152us) log.cc:844] Serialized 29 byte log entry
0504 13:56:32.097735 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:32.097793 (+ 58us) write_op.cc:454] Released schema lock
0504 13:56:32.097829 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:32.098017 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:32.098040 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":64538496,"child_traces":[["op",{"apply.queue_time_us":1483,"apply.run_cpu_time_us":1344,"apply.run_wall_time_us":1344,"num_ops":2,"prepare.queue_time_us":119,"prepare.run_cpu_time_us":2092,"prepare.run_wall_time_us":2088,"replication_time_us":74862}]]}
I20260504 13:56:32.186196 9518 tablet_service.cc:1558] Processing DeleteTablet for tablet 1501455a0af348e38746d100987afdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:32.186517 9515 tablet_service.cc:1558] Processing DeleteTablet for tablet 604f1ba5f71041a7877daff8972a994a with delete_type TABLET_DATA_DELETED (replaced by tablet 2ddf890c17964e7f90409ebe3c4034ef) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:32.186950 9520 tablet_service.cc:1558] Processing DeleteTablet for tablet 9ed4f5ea230a413a8dcdbb661ce173b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 88619eaa917d4c33a910fc1347ab126e) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:32.184051 9514 tablet_service.cc:1558] Processing DeleteTablet for tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:24 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:32.184048 9516 tablet_service.cc:1558] Processing DeleteTablet for tablet 50e129980cd84f6ca22cbf2addb06230 with delete_type TABLET_DATA_DELETED (replaced by tablet c47b6e3b4d0c45c28384f7aae6b749de) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:32.189356 10918 ts_tablet_manager.cc:1929] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:32.189754 10918 log.cc:1199] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/8bb77106b48a45a2aaa938dc474fa8bd
I20260504 13:56:32.190383 10918 ts_tablet_manager.cc:1950] T 8bb77106b48a45a2aaa938dc474fa8bd P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
W20260504 13:56:32.191987 10690 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.238.252:44781 timed out after 0.656s (SENT)
I20260504 13:56:32.172562 9931 ts_tablet_manager.cc:1403] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.152s user 0.012s sys 0.009s
I20260504 13:56:32.194567 9931 raft_consensus.cc:359] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.195398 9516 tablet_service.cc:1558] Processing DeleteTablet for tablet 8bb77106b48a45a2aaa938dc474fa8bd with delete_type TABLET_DATA_DELETED (replaced by tablet f337e9698fe64282a684e90d9c61da5a) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:32.195660 10814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 349 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7"
I20260504 13:56:32.196244 10814 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 1 FOLLOWER]: Advancing to term 2
I20260504 13:56:32.204509 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 2 candidate_status { last_received { term: 1 index: 349 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1"
I20260504 13:56:32.205953 10940 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 election: Requested vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:32.209487 10814 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 806407748d564439807fe5d73db9abda in term 2.
I20260504 13:56:32.210467 9173 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda; no voters: ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:32.213193 10681 raft_consensus.cc:493] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:32.213697 10681 raft_consensus.cc:515] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.216553 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a89bac060214d30924398007dfa1fa1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:32.217993 9620 leader_election.cc:343] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:32.218010 9198 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 2 FOLLOWER]: Rejecting Update request from peer ec9f4c3a43d84fc9a420dd10723c01a1 for earlier term 1. Current term is 2. Ops: []
I20260504 13:56:32.166503 9336 rpcz_store.cc:276] 0504 13:56:30.028094 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:31.963218 (+1935124us) service_pool.cc:224] Handling call
0504 13:56:32.011549 (+ 48331us) spinlock_profiling.cc:243] Waited 30.8 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6b0936 00007fc27b2bd524 00007fc27b2bd4b2 00007fc27b2bd3e1 00007fc27b2bd35c 00007fc27b2bc612 00007fc27958bbcc 00007fc27958b006
0504 13:56:32.013739 (+ 2190us) catalog_manager.cc:2101] Acquired catalog manager lock
0504 13:56:32.015131 (+ 1392us) catalog_manager.cc:2147] Created new table and tablet info
0504 13:56:32.165506 (+150375us) catalog_manager.cc:2207] Wrote table and tablets to system table
0504 13:56:32.165835 (+ 329us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0504 13:56:32.166129 (+ 294us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:32.093751 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:32.094171 (+ 420us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:32.094199 (+ 28us) write_op.cc:435] Acquired schema lock
0504 13:56:32.094214 (+ 15us) tablet.cc:662] Decoding operations
0504 13:56:32.094488 (+ 274us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:32.094523 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:32.094551 (+ 28us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:32.094751 (+ 200us) tablet.cc:701] Row locks acquired
0504 13:56:32.094767 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:32.094882 (+ 115us) write_op.cc:270] Start()
0504 13:56:32.094958 (+ 76us) write_op.cc:276] Timestamp: P: 1777902992094802 usec, L: 0
0504 13:56:32.094976 (+ 18us) op_driver.cc:348] REPLICATION: starting
0504 13:56:32.098872 (+ 3896us) log.cc:844] Serialized 433 byte log entry
0504 13:56:32.099512 (+ 640us) spinlock_profiling.cc:243] Waited 3.42 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b801951 00007fc27b894208 00007fc27bd069f1 00007fc27bd061c8 00007fc27bd0ae61 00007fc27bd0ae19 00007fc27bd0ada9 00007fc27bd0ad71 00007fc27bd09f9d 00000000005b3bc4 00000000005b39f9
0504 13:56:32.143440 (+ 43928us) op_driver.cc:464] REPLICATION: finished
0504 13:56:32.143790 (+ 350us) write_op.cc:301] APPLY: starting
0504 13:56:32.143856 (+ 66us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:32.143993 (+ 137us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:32.144023 (+ 30us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:32.144499 (+ 476us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:32.144689 (+ 190us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0504 13:56:32.144749 (+ 60us) write_op.cc:312] APPLY: finished
0504 13:56:32.144965 (+ 216us) log.cc:844] Serialized 29 byte log entry
0504 13:56:32.145155 (+ 190us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:32.145240 (+ 85us) write_op.cc:454] Released schema lock
0504 13:56:32.145281 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:32.145541 (+ 260us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:32.145575 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":70780672,"child_traces":[["op",{"apply.queue_time_us":180,"apply.run_cpu_time_us":1702,"apply.run_wall_time_us":1702,"num_ops":2,"prepare.queue_time_us":2212,"prepare.run_cpu_time_us":2559,"prepare.run_wall_time_us":5882,"replication_time_us":48359,"spinlock_wait_cycles":7877376}]]}
I20260504 13:56:32.219511 10681 leader_election.cc:290] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:32.222676 9932 tablet_bootstrap.cc:492] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:32.223065 9932 ts_tablet_manager.cc:1403] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.066s user 0.009s sys 0.005s
I20260504 13:56:32.224076 9509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6677a93d01d54f15a506ae1474bcb501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:32.226409 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:32.227347 9931 raft_consensus.cc:385] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:32.227691 9931 raft_consensus.cc:740] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:32.228433 9931 consensus_queue.cc:260] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.229915 10690 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:32.230382 9931 ts_tablet_manager.cc:1434] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.038s user 0.004s sys 0.000s
I20260504 13:56:32.232761 10940 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 2 FOLLOWER]: Leader election won for term 2
I20260504 13:56:32.233963 10945 consensus_queue.cc:1059] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 }, Status: INVALID_TERM, Last received: 1.349, Next index: 350, Last known committed idx: 348, Time since last communication: 0.001s
I20260504 13:56:32.234848 10945 raft_consensus.cc:3055] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 1 LEADER]: Stepping down as leader of term 1
I20260504 13:56:32.235242 10945 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec9f4c3a43d84fc9a420dd10723c01a1, State: Running, Role: LEADER
I20260504 13:56:32.236094 10945 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 1.350, Last appended by leader: 350, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:32.237375 10945 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 1 FOLLOWER]: Advancing to term 2
I20260504 13:56:32.238384 10890 raft_consensus.cc:2804] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:32.238823 10890 raft_consensus.cc:493] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:32.239141 10890 raft_consensus.cc:3060] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:32.151006 10777 ts_manager.cc:194] Registered new tserver with Master: d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:32.250074 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e70772b7ce4f27a941918aaf66a865. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:32.252245 9892 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.238.252:44781)
I20260504 13:56:32.259410 9329 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 8bb77106b48a45a2aaa938dc474fa8bd (table default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]) successfully deleted
I20260504 13:56:32.260890 9931 tablet_bootstrap.cc:492] T bae90898739b4303af0608e39e8c0367 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:32.268999 9508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f4c8ffb2175493a981c18676485ac04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:32.284219 10788 raft_consensus.cc:493] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:32.285075 10940 raft_consensus.cc:697] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 2 LEADER]: Becoming Leader. State: Replica: 806407748d564439807fe5d73db9abda, State: Running, Role: LEADER
I20260504 13:56:32.284806 10788 raft_consensus.cc:515] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.286151 10940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 1.349, Last appended by leader: 349, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
W20260504 13:56:32.290124 9768 leader_election.cc:343] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:32.287086 10788 leader_election.cc:290] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:32.292184 10962 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:32.293717 9932 raft_consensus.cc:359] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.294399 9932 raft_consensus.cc:385] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:32.305883 10963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 806407748d564439807fe5d73db9abda. Latest consensus state: current_term: 2 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:32.306722 10963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: This master's current role is: LEADER
I20260504 13:56:32.308213 9931 tablet_bootstrap.cc:654] T bae90898739b4303af0608e39e8c0367 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:32.309448 10690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.252:44781, user_credentials={real_user=slave}} blocked reactor thread for 79628.8us
I20260504 13:56:32.294638 9932 raft_consensus.cc:740] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
W20260504 13:56:32.321424 9892 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.238.252:44781)
W20260504 13:56:32.327489 9339 catalog_manager.cc:2848] Illegal state: an error occurred while updating the sys-catalog: replica ec9f4c3a43d84fc9a420dd10723c01a1 is not leader of this config: current role FOLLOWER
W20260504 13:56:32.329782 9892 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.8.238.252:44781)
W20260504 13:56:32.339452 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:32.346918 9932 consensus_queue.cc:260] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.384277 10960 raft_consensus.cc:493] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 13:56:32.377036 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 221817us
I20260504 13:56:32.384788 10960 raft_consensus.cc:515] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.389588 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47b6e3b4d0c45c28384f7aae6b749de" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:32.391291 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a629c2a69084b6b98984c8d374eeb1e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:32.392167 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a89bac060214d30924398007dfa1fa1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:32.387542 9401 catalog_manager.cc:1612] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260504 13:56:32.394387 9585 heartbeater.cc:344] Connected to a master server at 127.8.238.252:44781
I20260504 13:56:32.274852 10890 raft_consensus.cc:515] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.417906 10013 ts_tablet_manager.cc:1434] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 1.100s user 0.008s sys 0.019s
W20260504 13:56:32.418258 10013 ts_tablet_manager.cc:1467] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00: Tablet startup took 1100ms
I20260504 13:56:32.425069 9932 ts_tablet_manager.cc:1434] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.202s user 0.007s sys 0.000s
I20260504 13:56:32.428579 9680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bae90898739b4303af0608e39e8c0367" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:32.429100 9680 raft_consensus.cc:3060] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:32.387036 10960 leader_election.cc:290] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:32.436547 9680 raft_consensus.cc:2468] T bae90898739b4303af0608e39e8c0367 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
W20260504 13:56:32.447480 9618 leader_election.cc:343] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:32.448012 9618 leader_election.cc:304] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:32.449173 10970 raft_consensus.cc:2749] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:32.449168 9618 leader_election.cc:343] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:32.450151 9618 leader_election.cc:304] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:32.451042 9618 leader_election.cc:343] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:32.451416 9618 leader_election.cc:304] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:32.452131 10970 raft_consensus.cc:2749] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:32.452664 10970 raft_consensus.cc:2749] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:32.466121 9536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "184fefa651fc411fb5107cc923c02535" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:32.466929 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a985338790e4887af38c49ba013417f" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:32.472476 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60d3cc59ffe947fbb57b7d3301ccbac1" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:32.479192 9532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274bb204041c4e1085395c7bd938aaf1" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:32.485558 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41d41f8ba6dc4bb18116cb2b2dfabff0" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:32.486368 9530 raft_consensus.cc:2468] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:32.475052 10810 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 2 FOLLOWER]: Rejecting Update request from peer ec9f4c3a43d84fc9a420dd10723c01a1 for earlier term 1. Current term is 2. Ops: [1.5-1.336]
I20260504 13:56:32.503407 10890 leader_election.cc:290] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:32.506261 10788 raft_consensus.cc:493] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7b79fe7dd8934dd996393a2d96eabf00)
I20260504 13:56:32.506745 10788 raft_consensus.cc:515] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:32.508525 10788 leader_election.cc:290] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:32.509979 9680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fdc09a498be470ab04563c62caf7755" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:32.517844 10890 raft_consensus.cc:493] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:32.518277 10890 raft_consensus.cc:515] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.520090 10890 leader_election.cc:290] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:32.521598 10391 raft_consensus.cc:2243] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:32.525738 9678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fae9274bb8240798f85b79504a227b3" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:32.526288 9678 raft_consensus.cc:2468] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
W20260504 13:56:32.418534 10013 ts_tablet_manager.cc:1469] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00: Trace:
0504 13:56:23.716662 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:23.716781 (+ 119us) service_pool.cc:224] Handling call
0504 13:56:23.717957 (+ 1176us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:23.718032 (+ 75us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:23.844996 (+126964us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:23.974386 (+129390us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:23.974777 (+ 391us) inbound_call.cc:177] Queueing success response
0504 13:56:30.595381 (+6620604us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0504 13:56:31.317951 (+722570us) ts_tablet_manager.cc:1435] Starting tablet replica
0504 13:56:31.318335 (+ 384us) tablet_replica.cc:261] Starting instrumentation
0504 13:56:31.318782 (+ 447us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":433631,"raft.queue_time_us":20127,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":6002,"tablet-open.queue_time_us":6620692,"thread_start_us":164973,"threads_started":1}
I20260504 13:56:32.528262 10013 tablet_bootstrap.cc:492] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:32.535006 10681 raft_consensus.cc:493] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:32.535521 10681 raft_consensus.cc:515] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.537698 10681 leader_election.cc:290] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:32.540916 9680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ebc96408d64c2da7dc6c4fc158995c" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:32.541126 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d9bd2722d23420ea0eac7664357a60c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:32.541433 9680 raft_consensus.cc:3060] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:32.552001 9620 leader_election.cc:343] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:32.580659 10391 raft_consensus.cc:2272] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:32.580663 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d9bd2722d23420ea0eac7664357a60c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:32.583339 9618 leader_election.cc:343] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:32.583760 9618 leader_election.cc:304] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:32.584970 10681 raft_consensus.cc:2749] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:32.590282 9473 leader_election.cc:304] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
W20260504 13:56:32.594672 9473 consensus_peers.cc:597] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46fdf0ffaf454689a705949bd39deb26. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:32.595979 10956 raft_consensus.cc:2804] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:32.596154 9473 leader_election.cc:304] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:32.596495 10956 raft_consensus.cc:697] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:32.600421 10391 ts_tablet_manager.cc:1916] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:32.600786 10788 raft_consensus.cc:493] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:32.600526 10954 consensus_queue.cc:1048] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:32.601197 10788 raft_consensus.cc:515] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.602479 10954 raft_consensus.cc:2804] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:32.602800 10954 raft_consensus.cc:493] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:32.604863 10788 leader_election.cc:290] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:32.603147 10954 raft_consensus.cc:3060] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:32.605190 10956 consensus_queue.cc:237] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:32.606348 9474 leader_election.cc:343] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:32.613993 9474 consensus_peers.cc:597] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:32.561367 9680 raft_consensus.cc:2468] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
W20260504 13:56:32.614950 9474 consensus_peers.cc:597] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:32.621001 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68e92cc3da9b4bcfadb6fa26245f1c99" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:32.621690 9686 raft_consensus.cc:2393] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:32.624258 9931 tablet_bootstrap.cc:492] T bae90898739b4303af0608e39e8c0367 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:32.624724 9931 ts_tablet_manager.cc:1403] T bae90898739b4303af0608e39e8c0367 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.394s user 0.006s sys 0.004s
I20260504 13:56:32.627789 9931 raft_consensus.cc:359] T bae90898739b4303af0608e39e8c0367 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.657963 9932 tablet_bootstrap.cc:492] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:32.666134 9932 tablet_bootstrap.cc:654] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:32.661521 9474 consensus_peers.cc:597] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:32.675912 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 78976.1us
W20260504 13:56:32.679747 9474 leader_election.cc:343] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:32.685652 10960 consensus_queue.cc:1048] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:32.687443 10013 tablet_bootstrap.cc:654] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:32.695567 9931 raft_consensus.cc:385] T bae90898739b4303af0608e39e8c0367 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:32.697661 9507 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:52642 (request call id 244) took 1389 ms. Trace:
I20260504 13:56:32.698148 9507 rpcz_store.cc:276] 0504 13:56:31.197694 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:31.373936 (+176242us) service_pool.cc:224] Handling call
0504 13:56:31.457523 (+ 83587us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0504 13:56:31.457626 (+ 103us) ts_tablet_manager.cc:682] Creating new metadata...
0504 13:56:32.330505 (+872879us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:32.477481 (+146976us) tablet_replica.cc:206] Creating consensus instance
0504 13:56:32.587125 (+109644us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:32.701558 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bae90898739b4303af0608e39e8c0367" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:32.701716 9931 raft_consensus.cc:740] T bae90898739b4303af0608e39e8c0367 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:32.705679 10981 raft_consensus.cc:493] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:32.706631 10981 raft_consensus.cc:515] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.709651 9680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d63f665166466dba1e7e36c5e19506" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:32.710388 9680 raft_consensus.cc:2393] T e5d63f665166466dba1e7e36c5e19506 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:32.710418 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fae9274bb8240798f85b79504a227b3" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:32.711582 10981 leader_election.cc:290] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:32.718062 10945 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:32.718482 10945 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:32.720685 10945 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:56:32.722020 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 3 candidate_status { last_received { term: 1 index: 350 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:32.724860 10810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 3 candidate_status { last_received { term: 1 index: 350 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:32.721518 10954 raft_consensus.cc:515] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.704886 9931 consensus_queue.cc:260] T bae90898739b4303af0608e39e8c0367 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.733513 9828 raft_consensus.cc:3060] T bae90898739b4303af0608e39e8c0367 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:32.738992 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 62375.3us
W20260504 13:56:32.741357 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 60608.4us
I20260504 13:56:32.745335 10810 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec9f4c3a43d84fc9a420dd10723c01a1 in term 2.
I20260504 13:56:32.746802 10970 raft_consensus.cc:493] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:32.746928 9328 leader_election.cc:304] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7, ec9f4c3a43d84fc9a420dd10723c01a1; no voters: 806407748d564439807fe5d73db9abda
I20260504 13:56:32.747260 10970 raft_consensus.cc:515] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.747920 10945 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 13:56:32.748234 10945 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:32.748535 10945 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 2 FOLLOWER]: Advancing to term 3
I20260504 13:56:32.749115 10970 leader_election.cc:290] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:32.752205 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac3e36cfa4848a1ab015f89c56c2740" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:32.755792 9620 leader_election.cc:343] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:32.763218 9932 tablet_bootstrap.cc:492] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:32.763458 10810 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 2 FOLLOWER]: Refusing update from remote peer 806407748d564439807fe5d73db9abda: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 350. (index mismatch)
I20260504 13:56:32.763617 9932 ts_tablet_manager.cc:1403] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.131s user 0.008s sys 0.003s
I20260504 13:56:32.765095 10963 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [LEADER]: Connected to new peer: Peer: permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 13:56:32.766357 9932 raft_consensus.cc:359] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.766992 9932 raft_consensus.cc:385] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:32.767266 9932 raft_consensus.cc:740] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:32.767858 9932 consensus_queue.cc:260] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.752553 9828 raft_consensus.cc:2468] T bae90898739b4303af0608e39e8c0367 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:32.773847 10933 catalog_manager.cc:1269] Loaded cluster ID: 302d3e241578494980ac3509cf5306fb
I20260504 13:56:32.774092 10933 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7: loading cluster ID for follower catalog manager: success
I20260504 13:56:32.776721 9931 ts_tablet_manager.cc:1434] T bae90898739b4303af0608e39e8c0367 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.152s user 0.006s sys 0.001s
I20260504 13:56:32.777567 9931 tablet_bootstrap.cc:492] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:32.781028 9932 ts_tablet_manager.cc:1434] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260504 13:56:32.803437 10681 raft_consensus.cc:493] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:32.803380 10945 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:32.803869 10681 raft_consensus.cc:515] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.805586 10681 leader_election.cc:290] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:32.808904 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 3 candidate_status { last_received { term: 1 index: 350 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda"
I20260504 13:56:32.809195 10810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 3 candidate_status { last_received { term: 1 index: 350 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7"
I20260504 13:56:32.809856 10810 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 2 FOLLOWER]: Advancing to term 3
I20260504 13:56:32.812731 10933 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7: acquiring CA information for follower catalog manager: success
I20260504 13:56:32.817371 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cb30835150949f39f6e7e7563381763" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:32.819427 10810 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec9f4c3a43d84fc9a420dd10723c01a1 in term 3.
W20260504 13:56:32.820204 9620 leader_election.cc:343] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:32.820580 9328 leader_election.cc:304] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7, ec9f4c3a43d84fc9a420dd10723c01a1; no voters: 806407748d564439807fe5d73db9abda
I20260504 13:56:32.821189 9932 tablet_bootstrap.cc:492] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:32.824194 10933 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260504 13:56:32.833281 9931 tablet_bootstrap.cc:654] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:32.838883 9473 leader_election.cc:304] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:32.840304 9932 tablet_bootstrap.cc:654] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:32.841401 9532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cb30835150949f39f6e7e7563381763" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:32.845479 10791 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" instance_seqno: 1777902963144521) as {username='slave'} at 127.8.238.193:53697; Asking this server to re-register.
I20260504 13:56:32.848750 10961 consensus_queue.cc:1048] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260504 13:56:32.849498 10945 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 3 election: Requested vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:56:32.851856 10945 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 3 FOLLOWER]: Leader election won for term 3
W20260504 13:56:32.852350 9618 leader_election.cc:343] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:32.852744 9618 leader_election.cc:304] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:32.854099 10681 raft_consensus.cc:2749] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:32.854624 10945 raft_consensus.cc:697] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 3 LEADER]: Becoming Leader. State: Replica: ec9f4c3a43d84fc9a420dd10723c01a1, State: Running, Role: LEADER
I20260504 13:56:32.841791 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac3e36cfa4848a1ab015f89c56c2740" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:32.858254 9537 raft_consensus.cc:2468] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:32.860616 9587 heartbeater.cc:461] Registering TS with master...
I20260504 13:56:32.860694 9618 leader_election.cc:304] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:32.861896 10970 raft_consensus.cc:2804] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:32.862211 10970 raft_consensus.cc:493] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:32.862504 10970 raft_consensus.cc:3060] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:32.868032 10954 leader_election.cc:290] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:32.873301 10970 raft_consensus.cc:515] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.876704 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac3e36cfa4848a1ab015f89c56c2740" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:32.878319 9620 leader_election.cc:343] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:32.886632 10945 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 1.350, Last appended by leader: 350, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:32.893540 10970 leader_election.cc:290] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
W20260504 13:56:32.892239 9328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 71729.6us
W20260504 13:56:32.910048 9474 consensus_peers.cc:597] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46fdf0ffaf454689a705949bd39deb26. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:32.911976 9354 raft_consensus.cc:1240] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 3 LEADER]: Rejecting Update request from peer 806407748d564439807fe5d73db9abda for earlier term 2. Current term is 3. Ops: []
I20260504 13:56:32.842274 10955 consensus_queue.cc:1048] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.158s
I20260504 13:56:32.914856 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac3e36cfa4848a1ab015f89c56c2740" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:32.920288 9534 raft_consensus.cc:3060] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:32.920957 10996 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 }, Status: INVALID_TERM, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20260504 13:56:32.924324 10992 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec9f4c3a43d84fc9a420dd10723c01a1. Latest consensus state: current_term: 3 leader_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:32.925060 10992 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: This master's current role is: LEADER
I20260504 13:56:32.929347 9931 tablet_bootstrap.cc:492] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:32.929762 9931 ts_tablet_manager.cc:1403] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.152s user 0.007s sys 0.013s
I20260504 13:56:32.931946 9931 raft_consensus.cc:359] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.932529 9931 raft_consensus.cc:385] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:32.932761 9931 raft_consensus.cc:740] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:32.933490 9931 consensus_queue.cc:260] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.935385 9932 tablet_bootstrap.cc:492] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:32.935796 9932 ts_tablet_manager.cc:1403] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.115s user 0.008s sys 0.012s
I20260504 13:56:32.936134 10985 raft_consensus.cc:2804] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:32.936581 10985 raft_consensus.cc:697] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:32.936977 10013 tablet_bootstrap.cc:492] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:32.937350 10013 ts_tablet_manager.cc:1403] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.409s user 0.014s sys 0.019s
I20260504 13:56:32.938424 9932 raft_consensus.cc:359] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:32.917753 9474 consensus_peers.cc:597] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:32.939034 9932 raft_consensus.cc:385] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:32.939299 9932 raft_consensus.cc:740] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:32.939884 9932 consensus_queue.cc:260] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.937353 10985 consensus_queue.cc:237] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:32.946125 9474 leader_election.cc:343] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:32.949787 9931 ts_tablet_manager.cc:1434] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20260504 13:56:32.960399 9767 leader_election.cc:304] T 184fefa651fc411fb5107cc923c02535 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:32.961376 10981 raft_consensus.cc:2749] T 184fefa651fc411fb5107cc923c02535 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:32.963558 9767 leader_election.cc:304] T 8a985338790e4887af38c49ba013417f P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:32.965288 10981 raft_consensus.cc:2749] T 8a985338790e4887af38c49ba013417f P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:32.968004 9767 leader_election.cc:304] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:32.968119 10996 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [LEADER]: Peer ec9f4c3a43d84fc9a420dd10723c01a1 log is divergent from this leader: its last log entry 3.351 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 349
I20260504 13:56:32.969086 10788 raft_consensus.cc:2749] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:32.971705 9767 leader_election.cc:304] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:32.972404 10981 raft_consensus.cc:2749] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:32.973392 9767 leader_election.cc:304] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:32.974081 10981 raft_consensus.cc:2804] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:32.974397 10981 raft_consensus.cc:493] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:32.974678 10981 raft_consensus.cc:3060] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:32.976226 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6ebc96408d64c2da7dc6c4fc158995c" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:32.983765 9931 tablet_bootstrap.cc:492] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:32.985808 10970 raft_consensus.cc:493] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:32.986245 10970 raft_consensus.cc:515] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:32.988111 10970 leader_election.cc:290] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:32.989704 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "538c8c2412104c3e8b19394a1cc38818" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:32.991279 9620 leader_election.cc:343] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:32.991499 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac3e36cfa4848a1ab015f89c56c2740" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:32.994652 10997 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 2 LEADER]: Stepping down as leader of term 2
I20260504 13:56:32.995000 10997 raft_consensus.cc:740] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 806407748d564439807fe5d73db9abda, State: Running, Role: LEADER
I20260504 13:56:32.995817 10997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 2.350, Last appended by leader: 350, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:32.999506 9932 ts_tablet_manager.cc:1434] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.063s user 0.002s sys 0.004s
I20260504 13:56:33.001329 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "538c8c2412104c3e8b19394a1cc38818" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:33.002564 9618 leader_election.cc:343] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:33.008756 9618 leader_election.cc:304] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:33.009733 10970 raft_consensus.cc:2749] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:33.003063 10013 raft_consensus.cc:359] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.010192 10013 raft_consensus.cc:385] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:33.010454 10013 raft_consensus.cc:740] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:33.010978 10013 consensus_queue.cc:260] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.014680 10013 ts_tablet_manager.cc:1434] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.076s user 0.006s sys 0.000s
I20260504 13:56:33.015584 10013 tablet_bootstrap.cc:492] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:33.015969 9932 tablet_bootstrap.cc:492] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:33.020864 9534 raft_consensus.cc:2468] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:33.021826 9618 leader_election.cc:304] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:33.023106 10970 raft_consensus.cc:2804] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:33.023546 10970 raft_consensus.cc:697] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:33.024336 10970 consensus_queue.cc:237] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.027782 9344 master_service.cc:954] ReplaceTablet: received request to replace tablet 68e92cc3da9b4bcfadb6fa26245f1c99 from {username='slave'} at 127.0.0.1:35768
W20260504 13:56:33.029904 9892 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.238.252:44781)
I20260504 13:56:33.036948 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fae9274bb8240798f85b79504a227b3" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:33.037544 10997 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 2 FOLLOWER]: Advancing to term 3
I20260504 13:56:33.044099 9931 tablet_bootstrap.cc:654] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:32.914531 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 75709.8us
I20260504 13:56:33.060210 9932 tablet_bootstrap.cc:654] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:33.064985 9331 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5cb30835150949f39f6e7e7563381763 consensus state has no leader
W20260504 13:56:33.066846 9474 leader_election.cc:343] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:32.982874 10981 raft_consensus.cc:515] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:33.088311 9768 leader_election.cc:343] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:33.091421 10970 raft_consensus.cc:493] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:33.091995 10970 raft_consensus.cc:515] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.098022 10970 leader_election.cc:290] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:33.107020 9824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e53381ee0e7f4e9781b833b435bca831" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:33.108284 9620 leader_election.cc:343] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:33.111532 10777 ts_manager.cc:194] Registered new tserver with Master: 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:33.114423 10981 leader_election.cc:290] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:33.115720 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 65717.1us
I20260504 13:56:33.118551 9932 tablet_bootstrap.cc:492] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:33.118971 9932 ts_tablet_manager.cc:1403] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.103s user 0.008s sys 0.001s
W20260504 13:56:33.121939 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 54205.8us
I20260504 13:56:33.040275 10391 ts_tablet_manager.cc:1929] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:33.123224 10391 log.cc:1199] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:33.121584 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fdc09a498be470ab04563c62caf7755" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:33.124171 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68e92cc3da9b4bcfadb6fa26245f1c99" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:33.121635 9532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5d63f665166466dba1e7e36c5e19506" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:33.125448 9932 raft_consensus.cc:359] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.126032 10391 ts_tablet_manager.cc:1950] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:33.126080 9932 raft_consensus.cc:385] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:33.126291 9932 raft_consensus.cc:740] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:33.139971 9932 consensus_queue.cc:260] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.157908 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e53381ee0e7f4e9781b833b435bca831" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:33.172315 9618 leader_election.cc:343] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:33.172715 9618 leader_election.cc:304] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:33.174093 10970 raft_consensus.cc:2749] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:33.179028 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 56163.5us
W20260504 13:56:33.179804 9474 leader_election.cc:343] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:33.182507 10013 tablet_bootstrap.cc:654] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:33.188508 9932 ts_tablet_manager.cc:1434] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.069s user 0.007s sys 0.000s
I20260504 13:56:33.189507 9344 master_service.cc:954] ReplaceTablet: received request to replace tablet c6ebc96408d64c2da7dc6c4fc158995c from {username='slave'} at 127.0.0.1:35768
I20260504 13:56:33.192289 9932 tablet_bootstrap.cc:492] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:33.195415 9931 tablet_bootstrap.cc:492] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:33.195966 9931 ts_tablet_manager.cc:1403] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.212s user 0.013s sys 0.006s
I20260504 13:56:33.207341 9931 raft_consensus.cc:359] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.208042 9931 raft_consensus.cc:385] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:33.208312 9931 raft_consensus.cc:740] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:33.209299 9931 consensus_queue.cc:260] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.218904 9931 ts_tablet_manager.cc:1434] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260504 13:56:33.219780 9931 tablet_bootstrap.cc:492] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:33.221565 9474 leader_election.cc:343] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:33.224901 9767 leader_election.cc:304] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:33.226174 9767 leader_election.cc:304] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:33.227313 10981 raft_consensus.cc:2749] T 4fdc09a498be470ab04563c62caf7755 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:33.227856 10981 raft_consensus.cc:3060] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:33.227108 9767 leader_election.cc:304] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:33.254259 9932 tablet_bootstrap.cc:654] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:33.254906 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 138094us
I20260504 13:56:33.272285 9931 tablet_bootstrap.cc:654] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:33.279847 10788 raft_consensus.cc:3060] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:33.262971 10981 raft_consensus.cc:2749] T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:33.298811 10788 raft_consensus.cc:2749] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:33.303324 9175 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:33.305732 9932 tablet_bootstrap.cc:492] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:33.306221 9932 ts_tablet_manager.cc:1403] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.117s user 0.004s sys 0.004s
I20260504 13:56:33.310396 10814 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 3 FOLLOWER]: Refusing update from remote peer ec9f4c3a43d84fc9a420dd10723c01a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 351. (index mismatch)
I20260504 13:56:33.312966 10992 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 13:56:33.329169 9932 raft_consensus.cc:359] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.329785 9932 raft_consensus.cc:385] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:33.330035 9932 raft_consensus.cc:740] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:33.330619 9932 consensus_queue.cc:260] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.332576 9932 ts_tablet_manager.cc:1434] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
W20260504 13:56:33.339574 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 83623.2us
W20260504 13:56:33.303938 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 76877.2us
I20260504 13:56:33.344521 11010 raft_consensus.cc:493] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:33.344997 11010 raft_consensus.cc:515] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.332827 9931 tablet_bootstrap.cc:492] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:33.350956 11010 leader_election.cc:290] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:33.351619 11014 tablet.cc:1724] T 7cb6927537704cc4b17617cbdaa21ac1 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:33.352576 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac3e36cfa4848a1ab015f89c56c2740" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:33.353595 9678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fae9274bb8240798f85b79504a227b3" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:33.354166 9932 tablet_bootstrap.cc:492] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:33.354121 9678 raft_consensus.cc:3060] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:33.355341 9677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c12ffa802b840b1937f6f796083c478" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:33.365679 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 238) took 4164 ms. Trace:
I20260504 13:56:33.366037 10391 rpcz_store.cc:276] 0504 13:56:29.200954 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:29.201689 (+ 735us) service_pool.cc:224] Handling call
0504 13:56:31.700902 (+2499213us) ts_tablet_manager.cc:1151] Deleting tablet 41d41f8ba6dc4bb18116cb2b2dfabff0
0504 13:56:31.700958 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:32.596748 (+895790us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b28000dae08. stack: 00007f591ed51c83 00007f591ed51b80 00007f591e8c2574 000000000050e094 000000000050e029 00007f5927b7a251 00007f5927b8d7ea 00007f59261be16f 00007f592626b94b 00007f5926ec3cca 00007f5926ef4990 00007f5926edaefd 00007f5926edaea9 00007f5926edae39 00007f5926edae01 00007f5926eda02d
0504 13:56:32.941942 (+345194us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:33.040163 (+ 98221us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:33.178271 (+138108us) spinlock_profiling.cc:243] Waited 40.7 ms on lock 0x7b5800000098. stack: 00007f591ed51c83 00007f591ed51b80 00007f591e8c2574 000000000050e094 000000000050e029 00007f591ebc1941 00007f591ed466c1 00007f591ed46cf2 00007f5927bbc469 00007f5926dfc488 00007f5926ec3e12 00007f5926ef4990 00007f5926edaefd 00007f5926edaea9 00007f5926edae39 00007f5926edae01
0504 13:56:33.365601 (+187330us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":864428,"spinlock_wait_cycles":117166976,"tablet-delete.queue_time_us":2498682}
I20260504 13:56:33.350746 9931 ts_tablet_manager.cc:1403] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.131s user 0.009s sys 0.001s
I20260504 13:56:33.371322 9931 raft_consensus.cc:359] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.372108 9931 raft_consensus.cc:385] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:33.372349 9931 raft_consensus.cc:740] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:33.374007 9931 consensus_queue.cc:260] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.409926 9678 raft_consensus.cc:2468] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:33.413457 9931 ts_tablet_manager.cc:1434] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20260504 13:56:33.431710 9931 tablet_bootstrap.cc:492] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:33.433408 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 241) took 4037 ms. Trace:
I20260504 13:56:33.433782 10391 rpcz_store.cc:276] 0504 13:56:29.395746 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:29.395852 (+ 106us) service_pool.cc:224] Handling call
0504 13:56:33.367618 (+3971766us) ts_tablet_manager.cc:1151] Deleting tablet da61c91e10264fa49fe397d9a9b014bb
0504 13:56:33.427583 (+ 59965us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:33.433353 (+ 5770us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3971265}
I20260504 13:56:33.434789 10391 tablet_replica.cc:333] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:33.435652 10391 raft_consensus.cc:2243] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:33.436339 10391 raft_consensus.cc:2272] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:33.456960 10968 op_tracker.cc:284] OpTracker waiting for 1 outstanding ops to complete now for 1.021s
I20260504 13:56:33.457420 10968 op_tracker.cc:286] Dumping up to 50 currently running ops:
I20260504 13:56:33.457954 9931 tablet_bootstrap.cc:654] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:33.457639 10968 op_tracker.cc:290] R-P WriteOp [type=FOLLOWER, start_time=2026-05-04 13:56:32, state=WriteOpState 0x7b58000d4d00 [op_id=(term: 1 index: 349), ts=7282290655620308992, rows=[INSERT (int8 entry_type=<redacted>, string entry_id=<redacted>, string metadata=<redacted>), INSERT (int8 entry_type=<redacted>, string entry_id=<redacted>, string metadata=<redacted>)]]]
I20260504 13:56:33.463788 11021 tablet.cc:1724] T a1cf52a9e58744a99dae1958d2c1f867 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:33.411913 10681 raft_consensus.cc:493] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7b79fe7dd8934dd996393a2d96eabf00)
I20260504 13:56:33.469753 10681 raft_consensus.cc:515] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:33.471652 10681 leader_election.cc:290] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:33.447047 9932 tablet_bootstrap.cc:654] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:33.472761 9824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fdc09a498be470ab04563c62caf7755" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:33.473600 9473 leader_election.cc:304] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:33.485507 11033 tablet.cc:1724] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:33.491436 10391 ts_tablet_manager.cc:1916] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:33.496441 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c12ffa802b840b1937f6f796083c478" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:33.498407 10956 raft_consensus.cc:2749] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:33.509205 9931 tablet_bootstrap.cc:492] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:33.509626 9931 ts_tablet_manager.cc:1403] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.078s user 0.000s sys 0.009s
I20260504 13:56:33.492347 10681 raft_consensus.cc:493] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:33.510505 10681 raft_consensus.cc:515] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.512305 10681 leader_election.cc:290] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:33.514844 9932 tablet_bootstrap.cc:492] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:33.515313 9932 ts_tablet_manager.cc:1403] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.161s user 0.012s sys 0.000s
I20260504 13:56:33.518184 9932 raft_consensus.cc:359] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.518579 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "982f7c53a1f0482e862b53aeeee7cd25" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:33.518733 9932 raft_consensus.cc:385] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260504 13:56:33.519790 9620 leader_election.cc:343] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:33.519968 9931 raft_consensus.cc:359] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.520629 9931 raft_consensus.cc:385] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:33.520938 9931 raft_consensus.cc:740] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:33.526038 9931 consensus_queue.cc:260] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.527984 9931 ts_tablet_manager.cc:1434] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.018s user 0.005s sys 0.002s
I20260504 13:56:33.495604 11030 tablet.cc:1724] T 7cb6927537704cc4b17617cbdaa21ac1 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:33.541750 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fdc09a498be470ab04563c62caf7755" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:33.546784 9618 leader_election.cc:304] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:33.547770 10681 raft_consensus.cc:2749] T 4fdc09a498be470ab04563c62caf7755 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:33.549172 9473 leader_election.cc:304] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:33.550108 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "982f7c53a1f0482e862b53aeeee7cd25" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:33.555987 10956 raft_consensus.cc:2804] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
W20260504 13:56:33.556576 9767 leader_election.cc:343] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Not found: Tablet not found: 41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:33.556978 9767 leader_election.cc:304] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:33.557835 11010 raft_consensus.cc:2749] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260504 13:56:33.558233 9198 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 3 FOLLOWER]: Refusing update from remote peer ec9f4c3a43d84fc9a420dd10723c01a1: Log matching property violated. Preceding OpId in replica: term: 2 index: 350. Preceding OpId from leader: term: 3 index: 351. (index mismatch)
I20260504 13:56:33.560132 11035 tablet.cc:1724] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:33.559365 10956 raft_consensus.cc:697] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:33.562701 10013 tablet_bootstrap.cc:492] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:33.563083 10013 ts_tablet_manager.cc:1403] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.548s user 0.014s sys 0.014s
I20260504 13:56:33.563918 10970 raft_consensus.cc:493] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:33.561628 10956 consensus_queue.cc:237] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.564365 10970 raft_consensus.cc:515] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.566159 10970 leader_election.cc:290] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:33.566921 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 (table default.Table_0134ffe7086440ccbcadd076d8fe5d01 [id=693e9b6ed89b4ab7b9d5dc7402e17f20]) successfully deleted
W20260504 13:56:33.572788 9618 leader_election.cc:343] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:33.573205 9618 leader_election.cc:304] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:33.574246 10681 raft_consensus.cc:2749] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:33.575265 11037 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 348, Time since last communication: 0.000s
I20260504 13:56:33.575960 11037 consensus_queue.cc:1243] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [LEADER]: Peer 806407748d564439807fe5d73db9abda log is divergent from this leader: its last log entry 2.350 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 348
I20260504 13:56:33.578469 9198 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.348->[1.349-3.351] Dedup: 1.349->[1.350-3.351]
I20260504 13:56:33.578962 9198 pending_rounds.cc:85] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: Aborting all ops after (but not including) 349
I20260504 13:56:33.579223 9198 pending_rounds.cc:107] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: Aborting uncommitted NO_OP operation due to leader change: 2.350
I20260504 13:56:33.579420 9198 raft_consensus.cc:2889] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260504 13:56:33.518992 9932 raft_consensus.cc:740] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:33.565394 9767 leader_election.cc:304] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:33.583709 11010 raft_consensus.cc:2749] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:33.566726 11038 tablet.cc:1724] T a1cf52a9e58744a99dae1958d2c1f867 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:33.580698 9932 consensus_queue.cc:260] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.597020 9932 ts_tablet_manager.cc:1434] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.081s user 0.005s sys 0.000s
I20260504 13:56:33.613528 11033 tablet.cc:1724] T a1cf52a9e58744a99dae1958d2c1f867 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:33.614465 10810 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 3 FOLLOWER]: Rejecting Update request from peer 806407748d564439807fe5d73db9abda for earlier term 2. Current term is 3. Ops: [1.5-2.350]
I20260504 13:56:33.620020 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dffddd4b89984c54bc26dcb4a9488525" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:33.621548 9620 leader_election.cc:343] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:33.622128 10963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec9f4c3a43d84fc9a420dd10723c01a1. Latest consensus state: current_term: 3 leader_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:33.622805 10963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:56:33.636566 9931 tablet_bootstrap.cc:492] T 4cf1504f0202481ba8b08f9bec5bc3d9 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:33.657038 9932 tablet_bootstrap.cc:492] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:33.662405 11035 tablet.cc:1724] T 7cb6927537704cc4b17617cbdaa21ac1 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:33.663079 9932 tablet_bootstrap.cc:654] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:33.668632 9328 catalog_manager.cc:4982] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet da61c91e10264fa49fe397d9a9b014bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da61c91e10264fa49fe397d9a9b014bb already in progress: creating tablet
I20260504 13:56:33.672178 10963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:33.672821 10963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: This master's current role is: FOLLOWER
W20260504 13:56:33.673921 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 88727.4us
I20260504 13:56:33.677811 11031 tablet.cc:1724] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:33.702353 10970 raft_consensus.cc:493] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:33.703009 10970 raft_consensus.cc:515] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:33.704577 11014 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.352s user 0.000s sys 0.020s
I20260504 13:56:33.705245 10970 leader_election.cc:290] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:33.706832 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6103d953ace4cb8be3c580b3316f87c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:33.708530 9620 leader_election.cc:343] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:33.687942 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dffddd4b89984c54bc26dcb4a9488525" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:33.689098 10391 ts_tablet_manager.cc:1929] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:33.719199 10391 log.cc:1199] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/de639fda874248f0853e74ee60aaf758
I20260504 13:56:33.720062 10391 ts_tablet_manager.cc:1950] T de639fda874248f0853e74ee60aaf758 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:33.720072 10970 raft_consensus.cc:493] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:33.720636 10970 raft_consensus.cc:515] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.722986 10970 leader_election.cc:290] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:33.727574 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf1504f0202481ba8b08f9bec5bc3d9" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:33.728991 9620 leader_election.cc:343] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:33.738174 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 7eaa40eb4b1c425a9edfb7eaba953b65 (table default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e]). Expected version 1 got 0
I20260504 13:56:33.739974 9343 catalog_manager.cc:5671] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:33.746135 9932 tablet_bootstrap.cc:492] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:33.755924 10970 raft_consensus.cc:493] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:33.778004 10970 raft_consensus.cc:515] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.779827 10970 leader_election.cc:290] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:33.780503 9932 ts_tablet_manager.cc:1403] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.124s user 0.010s sys 0.000s
I20260504 13:56:33.781198 9826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92480654807d495b879576b0195871bd" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:33.782725 9620 leader_election.cc:343] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:33.782661 9932 raft_consensus.cc:359] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.815201 9932 raft_consensus.cc:385] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:33.815475 9932 raft_consensus.cc:740] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:33.816349 9932 consensus_queue.cc:260] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.783545 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 274bb204041c4e1085395c7bd938aaf1 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]). Expected version 4 got 0
I20260504 13:56:33.777020 11049 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:33.818193 11049 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: This master's current role is: LEADER
I20260504 13:56:33.819142 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:33.820251 9343 catalog_manager.cc:5671] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:33.822503 9932 ts_tablet_manager.cc:1434] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.042s user 0.003s sys 0.001s
I20260504 13:56:33.823027 9343 catalog_manager.cc:5671] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:33.643050 9931 tablet_bootstrap.cc:654] T 4cf1504f0202481ba8b08f9bec5bc3d9 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:33.853260 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 1c12ffa802b840b1937f6f796083c478 (table default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1]). Expected version 1 got 0
I20260504 13:56:33.854117 9343 catalog_manager.cc:5671] T e5d63f665166466dba1e7e36c5e19506 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:33.794922 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 256) took 1653 ms. Trace:
I20260504 13:56:33.796030 10013 raft_consensus.cc:359] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:33.863523 10013 raft_consensus.cc:385] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:33.863849 10013 raft_consensus.cc:740] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:33.863425 10391 rpcz_store.cc:276] 0504 13:56:32.131284 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:32.167300 (+ 36016us) service_pool.cc:224] Handling call
0504 13:56:33.434644 (+1267344us) ts_tablet_manager.cc:1151] Deleting tablet de639fda874248f0853e74ee60aaf758
0504 13:56:33.434696 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:33.592772 (+158076us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:33.688985 (+ 96213us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:33.784402 (+ 95417us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266468}
I20260504 13:56:33.864638 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 267) took 1703 ms. Trace:
I20260504 13:56:33.864462 10013 consensus_queue.cc:260] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:33.856544 11021 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:52642 (request call id 270) took 1661 ms. Trace:
I20260504 13:56:33.864948 10391 rpcz_store.cc:276] 0504 13:56:32.161039 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:32.168511 (+ 7472us) service_pool.cc:224] Handling call
0504 13:56:33.864405 (+1695894us) ts_tablet_manager.cc:1151] Deleting tablet 604f1ba5f71041a7877daff8972a994a
0504 13:56:33.864443 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:33.864590 (+ 147us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1695219}
I20260504 13:56:33.882041 10013 ts_tablet_manager.cc:1434] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.319s user 0.005s sys 0.001s
I20260504 13:56:33.887954 10013 tablet_bootstrap.cc:492] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:33.865281 11021 rpcz_store.cc:276] 0504 13:56:32.195608 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:32.195857 (+ 249us) service_pool.cc:224] Handling call
0504 13:56:33.856468 (+1660611us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:32.304739 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0504 13:56:32.310882 (+ 6143us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0504 13:56:32.310923 (+ 41us) alter_schema_op.cc:66] Acquired schema lock
0504 13:56:32.310985 (+ 62us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0504 13:56:32.311104 (+ 119us) alter_schema_op.cc:128] START. Timestamp: 1777902992310973
0504 13:56:32.311124 (+ 20us) op_driver.cc:348] REPLICATION: starting
0504 13:56:32.330693 (+ 19569us) log.cc:844] Serialized 173 byte log entry
0504 13:56:33.326859 (+996166us) op_driver.cc:464] REPLICATION: finished
0504 13:56:33.462422 (+135563us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0504 13:56:33.466447 (+ 4025us) log.cc:844] Serialized 5 byte log entry
0504 13:56:33.810417 (+343970us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:33.814371 (+ 3954us) log.cc:844] Serialized 14 byte log entry
0504 13:56:33.824657 (+ 10286us) alter_schema_op.cc:73] Released schema lock
0504 13:56:33.831809 (+ 7152us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":122839,"prepare.queue_time_us":107826,"prepare.run_cpu_time_us":15762,"prepare.run_wall_time_us":435653,"replication_time_us":945261,"thread_start_us":589562,"threads_started":3,"wal-append.queue_time_us":398774}]]}
W20260504 13:56:33.898571 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 216608us
I20260504 13:56:33.866009 10391 tablet_replica.cc:333] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
W20260504 13:56:33.901281 9618 leader_election.cc:343] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:33.901449 11031 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.222s user 0.012s sys 0.003s
I20260504 13:56:33.901672 9618 leader_election.cc:304] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:33.902613 10681 raft_consensus.cc:2749] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:33.908851 9932 tablet_bootstrap.cc:492] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:33.911828 10968 catalog_manager.cc:1432] Term changed from 2 to 3 while waiting for master leader catchup. Not loading sys catalog metadata
I20260504 13:56:33.921636 10013 tablet_bootstrap.cc:654] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:33.922499 10391 raft_consensus.cc:2243] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:33.921749 9932 tablet_bootstrap.cc:654] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:33.866587 9328 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet de639fda874248f0853e74ee60aaf758 (table default.Table_0d4595642a604b8f8f57d3eadbbacacb [id=666099fdb1a94091ac6fd99cf58f18e2]) successfully deleted
W20260504 13:56:33.935485 9328 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 68927.6us
W20260504 13:56:33.936554 9328 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 604f1ba5f71041a7877daff8972a994a because the tablet was not found. No further retry: Not found: Tablet not found: 604f1ba5f71041a7877daff8972a994a
I20260504 13:56:33.940275 9343 catalog_manager.cc:5671] T 184fefa651fc411fb5107cc923c02535 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:33.942291 9343 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]). Expected version 2 got 0
I20260504 13:56:33.945326 9931 tablet_bootstrap.cc:492] T 4cf1504f0202481ba8b08f9bec5bc3d9 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:33.945283 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6103d953ace4cb8be3c580b3316f87c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:33.945962 9931 ts_tablet_manager.cc:1403] T 4cf1504f0202481ba8b08f9bec5bc3d9 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.310s user 0.005s sys 0.006s
I20260504 13:56:33.945384 10391 raft_consensus.cc:2272] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:33.948451 10681 raft_consensus.cc:493] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:33.948982 10681 raft_consensus.cc:515] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.950875 10681 leader_election.cc:290] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:33.953095 9826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0b9e47b0a4443c5830422b9d0553446" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:33.954469 9620 leader_election.cc:343] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:33.949787 11014 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:52642 (request call id 253) took 2548 ms. Trace:
I20260504 13:56:33.960956 9932 tablet_bootstrap.cc:492] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:33.961302 9932 ts_tablet_manager.cc:1403] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.006s
I20260504 13:56:33.963559 9932 raft_consensus.cc:359] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.964141 9932 raft_consensus.cc:385] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:33.964366 9932 raft_consensus.cc:740] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:33.965101 9932 consensus_queue.cc:260] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.968042 10391 ts_tablet_manager.cc:1916] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:33.968655 9932 ts_tablet_manager.cc:1434] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260504 13:56:33.969429 9932 tablet_bootstrap.cc:492] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:33.974905 9932 tablet_bootstrap.cc:654] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:33.978708 10890 raft_consensus.cc:493] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:33.981068 9931 raft_consensus.cc:359] T 4cf1504f0202481ba8b08f9bec5bc3d9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.981678 9931 raft_consensus.cc:385] T 4cf1504f0202481ba8b08f9bec5bc3d9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:33.981928 9931 raft_consensus.cc:740] T 4cf1504f0202481ba8b08f9bec5bc3d9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:33.982534 9931 consensus_queue.cc:260] T 4cf1504f0202481ba8b08f9bec5bc3d9 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.979118 10890 raft_consensus.cc:515] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:33.984556 9931 ts_tablet_manager.cc:1434] T 4cf1504f0202481ba8b08f9bec5bc3d9 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.038s user 0.004s sys 0.000s
I20260504 13:56:33.984730 10890 leader_election.cc:290] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:33.990900 9679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf1504f0202481ba8b08f9bec5bc3d9" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:33.991473 9679 raft_consensus.cc:2468] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:33.997083 9826 raft_consensus.cc:3060] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:33.994669 10967 consensus_queue.cc:1048] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260504 13:56:34.005669 9931 tablet_bootstrap.cc:492] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:34.003608 9473 leader_election.cc:304] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:34.008720 11060 raft_consensus.cc:2804] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:34.009047 11060 raft_consensus.cc:493] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:34.009330 11060 raft_consensus.cc:3060] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:33.959487 11014 rpcz_store.cc:276] 0504 13:56:31.395923 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:31.400610 (+ 4687us) service_pool.cc:224] Handling call
0504 13:56:33.940106 (+2539496us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:31.425871 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0504 13:56:31.426263 (+ 392us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0504 13:56:31.426293 (+ 30us) alter_schema_op.cc:66] Acquired schema lock
0504 13:56:31.426347 (+ 54us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0504 13:56:31.426459 (+ 112us) alter_schema_op.cc:128] START. Timestamp: 1777902991426393
0504 13:56:31.426480 (+ 21us) op_driver.cc:348] REPLICATION: starting
0504 13:56:31.427030 (+ 550us) log.cc:844] Serialized 173 byte log entry
0504 13:56:33.242756 (+1815726us) op_driver.cc:464] REPLICATION: finished
0504 13:56:33.351388 (+108632us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0504 13:56:33.352855 (+ 1467us) log.cc:844] Serialized 5 byte log entry
0504 13:56:33.466505 (+113650us) spinlock_profiling.cc:243] Waited 7.72 ms on lock 0x7b440003768c. stack: 00007f591ed51c83 00007f591ed51b80 00007f591e8c2574 000000000050e094 000000000050e029 00007f591ebc1941 00007f591ec90375 00007f591ec90d92 00007f5925c2257b 00007f5925c1de4f 00007f591ed6682c 00000000005a3e39 00007f591ed82cdb 00007f591ed80f7f 00007f591ed83242 00007f5925c105ab
0504 13:56:33.894807 (+428302us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:33.939858 (+ 45051us) log.cc:844] Serialized 14 byte log entry
0504 13:56:33.939963 (+ 105us) alter_schema_op.cc:73] Released schema lock
0504 13:56:33.939995 (+ 32us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":108406,"prepare.queue_time_us":14586,"prepare.run_cpu_time_us":3651,"prepare.run_wall_time_us":7699,"raft.queue_time_us":231,"raft.run_cpu_time_us":775,"raft.run_wall_time_us":775,"replication_time_us":1816133,"spinlock_wait_cycles":17754496,"thread_start_us":443089,"threads_started":3,"wal-append.queue_time_us":354323,"wal-append.run_cpu_time_us":1326,"wal-append.run_wall_time_us":1009650}]]}
I20260504 13:56:34.032023 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf1504f0202481ba8b08f9bec5bc3d9" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:34.032701 9828 raft_consensus.cc:2468] T 4cf1504f0202481ba8b08f9bec5bc3d9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:34.036546 10681 raft_consensus.cc:493] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:34.037024 10681 raft_consensus.cc:515] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.039744 10681 leader_election.cc:290] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:34.040437 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89edd71cf5aa4a6abe2084695d3ed807" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:34.041719 9620 leader_election.cc:343] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:34.048944 11066 log.cc:927] Time spent T e5d63f665166466dba1e7e36c5e19506 P d4051d2bf0bf410eb8acd107560df888: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20260504 13:56:34.060395 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92480654807d495b879576b0195871bd" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:34.048317 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf1504f0202481ba8b08f9bec5bc3d9" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:34.053691 10967 consensus_queue.cc:1048] T bae90898739b4303af0608e39e8c0367 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260504 13:56:34.065568 9828 raft_consensus.cc:3060] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:34.069948 10013 tablet_bootstrap.cc:492] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:34.070740 10013 ts_tablet_manager.cc:1403] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.183s user 0.005s sys 0.008s
I20260504 13:56:34.072520 11031 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:52642 (request call id 268) took 1911 ms. Trace:
W20260504 13:56:34.074023 9328 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet a862a43c5ac443d9bac98e0472ca23b2 consensus state has no leader
I20260504 13:56:34.076546 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0b9e47b0a4443c5830422b9d0553446" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:34.078263 10013 raft_consensus.cc:359] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.079419 10013 raft_consensus.cc:385] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:34.079790 10013 raft_consensus.cc:740] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:34.080502 9932 tablet_bootstrap.cc:492] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:34.080758 9342 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 274bb204041c4e1085395c7bd938aaf1 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]). Expected version 4 got 0
I20260504 13:56:34.080926 9932 ts_tablet_manager.cc:1403] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.112s user 0.011s sys 0.001s
I20260504 13:56:34.080935 10013 consensus_queue.cc:260] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.078233 11060 raft_consensus.cc:515] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.082337 9342 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:34.083514 9342 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]). Expected version 2 got 0
I20260504 13:56:34.084602 9342 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38]). Expected version 1 got 0
I20260504 13:56:34.085318 9931 tablet_bootstrap.cc:654] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:34.086302 9342 catalog_manager.cc:5671] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:34.090764 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89edd71cf5aa4a6abe2084695d3ed807" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:34.091550 9618 leader_election.cc:343] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:34.092042 9618 leader_election.cc:304] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:34.093083 10681 raft_consensus.cc:2749] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:34.094444 9618 leader_election.cc:343] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:34.094830 9618 leader_election.cc:304] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:34.095671 10681 raft_consensus.cc:2749] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:34.096539 9618 leader_election.cc:343] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:34.096962 9618 leader_election.cc:304] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:34.097831 10681 raft_consensus.cc:2749] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:34.100478 9147 master-stress-test.cc:577] Killing test thread 1/10
I20260504 13:56:34.100943 9342 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 7eaa40eb4b1c425a9edfb7eaba953b65 (table default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e]). Expected version 1 got 0
I20260504 13:56:34.102851 9342 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 1c12ffa802b840b1937f6f796083c478 (table default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1]). Expected version 1 got 0
I20260504 13:56:34.105773 10013 ts_tablet_manager.cc:1434] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.035s user 0.000s sys 0.007s
I20260504 13:56:34.106340 9342 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet de6714ee86d64112baf13dd1b2b87582 (table default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469]). Expected version 1 got 0
I20260504 13:56:34.108260 9335 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:35768 (request call id 279) took 4507 ms. Trace:
W20260504 13:56:34.108273 9618 leader_election.cc:343] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:34.108767 9618 leader_election.cc:304] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:34.110414 11026 consensus_queue.cc:1048] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:34.110702 10681 raft_consensus.cc:2749] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:34.108532 9335 rpcz_store.cc:276] 0504 13:56:29.600514 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:31.638867 (+2038353us) service_pool.cc:224] Handling call
0504 13:56:31.639708 (+ 841us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:31.640997 (+ 1289us) catalog_manager.cc:3725] Apply alter schema
0504 13:56:31.773640 (+132643us) catalog_manager.cc:3757] Acquired catalog manager lock
0504 13:56:31.773880 (+ 240us) catalog_manager.cc:3915] Updating metadata on disk
0504 13:56:32.069886 (+296006us) catalog_manager.cc:3946] Committing alterations to in-memory state
0504 13:56:34.108204 (+2038318us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:31.786188 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:31.786646 (+ 458us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:31.786673 (+ 27us) write_op.cc:435] Acquired schema lock
0504 13:56:31.786689 (+ 16us) tablet.cc:662] Decoding operations
0504 13:56:31.786960 (+ 271us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:31.786994 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:31.787018 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:31.787202 (+ 184us) tablet.cc:701] Row locks acquired
0504 13:56:31.787218 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:31.787336 (+ 118us) write_op.cc:270] Start()
0504 13:56:31.787423 (+ 87us) write_op.cc:276] Timestamp: P: 1777902991787284 usec, L: 0
0504 13:56:31.787443 (+ 20us) op_driver.cc:348] REPLICATION: starting
0504 13:56:31.787971 (+ 528us) log.cc:844] Serialized 405 byte log entry
0504 13:56:32.067558 (+279587us) op_driver.cc:464] REPLICATION: finished
0504 13:56:32.068034 (+ 476us) write_op.cc:301] APPLY: starting
0504 13:56:32.068096 (+ 62us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:32.068202 (+ 106us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:32.068303 (+ 101us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:32.068507 (+ 204us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:32.068658 (+ 151us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:32.068716 (+ 58us) write_op.cc:312] APPLY: finished
0504 13:56:32.068917 (+ 201us) log.cc:844] Serialized 23 byte log entry
0504 13:56:32.069591 (+ 674us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:32.069658 (+ 67us) write_op.cc:454] Released schema lock
0504 13:56:32.069698 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:32.070698 (+ 1000us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:32.070724 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":311,"apply.run_cpu_time_us":1386,"apply.run_wall_time_us":2622,"mutex_wait_us":3228,"num_ops":1,"prepare.queue_time_us":3332,"prepare.run_cpu_time_us":2003,"prepare.run_wall_time_us":2908,"replication_time_us":280036}]]}
I20260504 13:56:34.112095 10681 consensus_queue.cc:1048] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260504 13:56:34.125754 9932 raft_consensus.cc:359] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.126480 9932 raft_consensus.cc:385] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:34.072916 11031 rpcz_store.cc:276] 0504 13:56:32.161422 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:32.184676 (+ 23254us) service_pool.cc:224] Handling call
0504 13:56:34.072464 (+1887788us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:32.243266 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0504 13:56:32.243658 (+ 392us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0504 13:56:32.243703 (+ 45us) alter_schema_op.cc:66] Acquired schema lock
0504 13:56:32.243772 (+ 69us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0504 13:56:32.243887 (+ 115us) alter_schema_op.cc:128] START. Timestamp: 1777902992243755
0504 13:56:32.243917 (+ 30us) op_driver.cc:348] REPLICATION: starting
0504 13:56:32.244567 (+ 650us) log.cc:844] Serialized 173 byte log entry
0504 13:56:33.473884 (+1229317us) op_driver.cc:464] REPLICATION: finished
0504 13:56:33.677606 (+203722us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0504 13:56:33.679128 (+ 1522us) log.cc:844] Serialized 5 byte log entry
0504 13:56:34.022291 (+343163us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:34.071497 (+ 49206us) log.cc:844] Serialized 14 byte log entry
0504 13:56:34.072334 (+ 837us) alter_schema_op.cc:73] Released schema lock
0504 13:56:34.072363 (+ 29us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":203530,"prepare.queue_time_us":30560,"prepare.run_cpu_time_us":49938,"prepare.run_wall_time_us":188816,"replication_time_us":1229844,"thread_start_us":558116,"threads_started":3,"wal-append.queue_time_us":384621,"wal-append.run_cpu_time_us":1538,"wal-append.run_wall_time_us":1006039}]]}
I20260504 13:56:34.126777 9932 raft_consensus.cc:740] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:34.128281 9533 raft_consensus.cc:2393] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5beceb6317444247b31b44a6581707c6 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:34.131973 10788 raft_consensus.cc:493] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:34.132404 11060 leader_election.cc:290] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:34.132413 10788 raft_consensus.cc:515] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.133559 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf1504f0202481ba8b08f9bec5bc3d9" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:34.134050 9686 raft_consensus.cc:3060] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:34.135742 10788 leader_election.cc:290] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:34.136468 9678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac3e36cfa4848a1ab015f89c56c2740" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:34.138266 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf1504f0202481ba8b08f9bec5bc3d9" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:34.138792 9832 raft_consensus.cc:3060] T 4cf1504f0202481ba8b08f9bec5bc3d9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:34.140647 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fac3e36cfa4848a1ab015f89c56c2740" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:34.141321 9530 raft_consensus.cc:2393] T fac3e36cfa4848a1ab015f89c56c2740 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 5beceb6317444247b31b44a6581707c6 in this term.
I20260504 13:56:34.143788 9686 raft_consensus.cc:2468] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:34.144861 9473 leader_election.cc:304] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:34.145083 9767 leader_election.cc:304] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:34.145638 10956 raft_consensus.cc:2804] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:34.146135 10956 raft_consensus.cc:697] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:34.146664 11010 raft_consensus.cc:3060] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:34.146986 10956 consensus_queue.cc:237] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.154853 11010 raft_consensus.cc:2749] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:34.159286 9338 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 274bb204041c4e1085395c7bd938aaf1 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]). Expected version 4 got 0
I20260504 13:56:34.160352 9338 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]). Expected version 2 got 0
I20260504 13:56:34.162050 9618 leader_election.cc:304] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:34.163139 10681 raft_consensus.cc:2749] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:34.171784 10013 tablet_bootstrap.cc:492] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:34.173753 9932 consensus_queue.cc:260] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.186218 9338 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 7eaa40eb4b1c425a9edfb7eaba953b65 (table default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e]). Expected version 1 got 0
I20260504 13:56:34.192987 9932 ts_tablet_manager.cc:1434] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.112s user 0.008s sys 0.000s
I20260504 13:56:34.193887 9932 tablet_bootstrap.cc:492] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:34.199893 9338 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]). Expected version 2 got 0
I20260504 13:56:34.203799 9342 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee] alter complete (version 1)
I20260504 13:56:34.202358 9826 raft_consensus.cc:3060] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:34.211465 9341 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62] alter complete (version 1)
I20260504 13:56:34.214608 9832 raft_consensus.cc:2468] T 4cf1504f0202481ba8b08f9bec5bc3d9 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:34.226789 10013 tablet_bootstrap.cc:654] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:34.227200 10681 raft_consensus.cc:493] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:34.230516 10681 raft_consensus.cc:515] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.233695 10681 leader_election.cc:290] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:34.236263 10681 raft_consensus.cc:493] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:34.236711 10681 raft_consensus.cc:515] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.241698 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba4584755aae47e0b0f14c6c55474d18" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:34.244302 9340 catalog_manager.cc:5958] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Sending DeleteTablet for 3 replicas of tablet 60d3cc59ffe947fbb57b7d3301ccbac1
I20260504 13:56:34.245810 9340 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35768 (request call id 288) took 2073 ms. Trace:
I20260504 13:56:34.247525 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e86027e3e74bc196e9a9ffc9fdf8a5" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:34.246079 9340 rpcz_store.cc:276] 0504 13:56:32.172218 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:32.172341 (+ 123us) service_pool.cc:224] Handling call
0504 13:56:32.173529 (+ 1188us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0504 13:56:32.174655 (+ 1126us) catalog_manager.cc:2815] Modifying in-memory table state
0504 13:56:32.174802 (+ 147us) catalog_manager.cc:2824] Locking tablets
0504 13:56:32.175549 (+ 747us) catalog_manager.cc:2838] Removing table and tablets from system table
0504 13:56:33.727310 (+1551761us) catalog_manager.cc:2856] Removing table from by-name map
0504 13:56:33.727500 (+ 190us) catalog_manager.cc:2872] Committing in-memory state
0504 13:56:34.244191 (+516691us) catalog_manager.cc:2876] Aborting table tasks
0504 13:56:34.245756 (+ 1565us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:32.188052 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:32.188600 (+ 548us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:32.188631 (+ 31us) write_op.cc:435] Acquired schema lock
0504 13:56:32.188648 (+ 17us) tablet.cc:662] Decoding operations
0504 13:56:32.189031 (+ 383us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:32.189087 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:32.189113 (+ 26us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:32.189408 (+ 295us) tablet.cc:701] Row locks acquired
0504 13:56:32.189427 (+ 19us) write_op.cc:260] PREPARE: finished
0504 13:56:32.189551 (+ 124us) write_op.cc:270] Start()
0504 13:56:32.189646 (+ 95us) write_op.cc:276] Timestamp: P: 1777902992189502 usec, L: 0
0504 13:56:32.189666 (+ 20us) op_driver.cc:348] REPLICATION: starting
0504 13:56:32.190231 (+ 565us) log.cc:844] Serialized 796 byte log entry
0504 13:56:33.664905 (+1474674us) op_driver.cc:464] REPLICATION: finished
0504 13:56:33.717125 (+ 52220us) write_op.cc:301] APPLY: starting
0504 13:56:33.717204 (+ 79us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:33.717336 (+ 132us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:33.717456 (+ 120us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:33.717820 (+ 364us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:33.718022 (+ 202us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:33.718083 (+ 61us) write_op.cc:312] APPLY: finished
0504 13:56:33.718322 (+ 239us) log.cc:844] Serialized 29 byte log entry
0504 13:56:33.718497 (+ 175us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:33.718637 (+ 140us) write_op.cc:454] Released schema lock
0504 13:56:33.718678 (+ 41us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":52012,"apply.run_cpu_time_us":1867,"apply.run_wall_time_us":56498,"num_ops":2,"prepare.queue_time_us":10541,"prepare.run_cpu_time_us":2400,"prepare.run_wall_time_us":3649,"replication_time_us":1475141,"thread_start_us":61467,"threads_started":1}]]}
W20260504 13:56:34.259188 9341 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:55533 (request call id 47) took 3603 ms (client timeout 1000 ms). Trace:
I20260504 13:56:34.263638 11087 raft_consensus.cc:493] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:34.265199 11080 raft_consensus.cc:493] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:34.265661 11080 raft_consensus.cc:515] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:34.267393 9795 tablet_service.cc:1558] Processing DeleteTablet for tablet 60d3cc59ffe947fbb57b7d3301ccbac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:32 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:34.267802 9797 tablet_service.cc:1558] Processing DeleteTablet for tablet 46fdf0ffaf454689a705949bd39deb26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:28 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:34.268589 9801 tablet_service.cc:1558] Processing DeleteTablet for tablet 930078a10afa4dd987ad021cf9609514 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:22 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:34.267946 9799 tablet_service.cc:1558] Processing DeleteTablet for tablet de639fda874248f0853e74ee60aaf758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:28 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:34.269438 9799 tablet_service.cc:1558] Processing DeleteTablet for tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:24 UTC) from {username='slave'} at 127.0.0.1:56380
I20260504 13:56:34.269671 9801 tablet_service.cc:1558] Processing DeleteTablet for tablet 7c9030802900499a90c4dbaf5c6936a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6dd139c347f7406aac340cc6ac994273) from {username='slave'} at 127.0.0.1:56380
W20260504 13:56:34.259528 9341 rpcz_store.cc:269] 0504 13:56:30.655736 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:31.093407 (+437671us) service_pool.cc:224] Handling call
0504 13:56:34.259109 (+3165702us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:31.501793 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:31.502275 (+ 482us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:31.502303 (+ 28us) write_op.cc:435] Acquired schema lock
0504 13:56:31.502316 (+ 13us) tablet.cc:662] Decoding operations
0504 13:56:31.502601 (+ 285us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:31.502646 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:31.502671 (+ 25us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:31.502827 (+ 156us) tablet.cc:701] Row locks acquired
0504 13:56:31.502840 (+ 13us) write_op.cc:260] PREPARE: finished
0504 13:56:31.502974 (+ 134us) write_op.cc:270] Start()
0504 13:56:31.503071 (+ 97us) write_op.cc:276] Timestamp: P: 1777902991502900 usec, L: 0
0504 13:56:31.503092 (+ 21us) op_driver.cc:348] REPLICATION: starting
0504 13:56:31.512374 (+ 9282us) log.cc:844] Serialized 533 byte log entry
0504 13:56:31.939643 (+427269us) op_driver.cc:464] REPLICATION: finished
0504 13:56:31.985926 (+ 46283us) write_op.cc:301] APPLY: starting
0504 13:56:31.985980 (+ 54us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:31.986080 (+ 100us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:31.986194 (+ 114us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:31.986335 (+ 141us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:31.986467 (+ 132us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:31.986522 (+ 55us) write_op.cc:312] APPLY: finished
0504 13:56:31.986757 (+ 235us) log.cc:844] Serialized 23 byte log entry
0504 13:56:31.991353 (+ 4596us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:31.991413 (+ 60us) write_op.cc:454] Released schema lock
0504 13:56:31.991448 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:31.995126 (+ 3678us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:31.995152 (+ 26us) write_op.cc:454] Released schema lock
Related trace 'op':
0504 13:56:32.066223 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:32.066767 (+ 544us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:32.066805 (+ 38us) write_op.cc:435] Acquired schema lock
0504 13:56:32.066829 (+ 24us) tablet.cc:662] Decoding operations
0504 13:56:32.067136 (+ 307us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:32.067193 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:32.067221 (+ 28us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:32.067401 (+ 180us) tablet.cc:701] Row locks acquired
0504 13:56:32.067417 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:32.067561 (+ 144us) write_op.cc:270] Start()
0504 13:56:32.067678 (+ 117us) write_op.cc:276] Timestamp: P: 1777902992067469 usec, L: 0
0504 13:56:32.067697 (+ 19us) op_driver.cc:348] REPLICATION: starting
0504 13:56:32.070923 (+ 3226us) log.cc:844] Serialized 356 byte log entry
0504 13:56:32.079748 (+ 8825us) spinlock_profiling.cc:243] Waited 2.73 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b801951 00007fc27b894208 00007fc27bd069f1 00007fc27bd061c8 00007fc27bd0ae61 00007fc27bd0ae19 00007fc27bd0ada9 00007fc27bd0ad71 00007fc27bd09f9d 00000000005b3bc4 00000000005b39f9
0504 13:56:32.104914 (+ 25166us) op_driver.cc:464] REPLICATION: finished
0504 13:56:32.105389 (+ 475us) write_op.cc:301] APPLY: starting
0504 13:56:32.105439 (+ 50us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:32.105526 (+ 87us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:32.105545 (+ 19us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:32.105736 (+ 191us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:32.105864 (+ 128us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:32.105914 (+ 50us) write_op.cc:312] APPLY: finished
0504 13:56:32.106059 (+ 145us) log.cc:844] Serialized 23 byte log entry
0504 13:56:32.106191 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:32.106245 (+ 54us) write_op.cc:454] Released schema lock
0504 13:56:32.106273 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:32.106474 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:32.106499 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":46105,"apply.run_cpu_time_us":1336,"apply.run_wall_time_us":9175,"num_ops":1,"prepare.queue_time_us":110262,"prepare.run_cpu_time_us":2209,"prepare.run_wall_time_us":25344,"raft.queue_time_us":1137,"raft.run_cpu_time_us":576,"raft.run_wall_time_us":576,"replication_time_us":436442,"thread_start_us":46635,"threads_started":1}],["op",{"apply.queue_time_us":342,"apply.run_cpu_time_us":1049,"apply.run_wall_time_us":1048,"num_ops":1,"prepare.queue_time_us":81,"prepare.run_cpu_time_us":3820,"prepare.run_wall_time_us":13633,"replication_time_us":37130,"spinlock_wait_cycles":6318336}]]}
I20260504 13:56:34.284274 9931 tablet_bootstrap.cc:492] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:34.284910 9931 ts_tablet_manager.cc:1403] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.279s user 0.005s sys 0.011s
I20260504 13:56:34.287282 9931 raft_consensus.cc:359] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.291947 11087 raft_consensus.cc:515] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:34.293064 9343 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:38691 (request call id 45) took 3563 ms (client timeout 1000 ms). Trace:
I20260504 13:56:34.294171 11087 leader_election.cc:290] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:34.267702 10681 leader_election.cc:290] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
W20260504 13:56:34.293450 9343 rpcz_store.cc:269] 0504 13:56:30.729540 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:31.307148 (+577608us) service_pool.cc:224] Handling call
0504 13:56:34.293005 (+2985857us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:34.025395 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:34.026014 (+ 619us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:34.026053 (+ 39us) write_op.cc:435] Acquired schema lock
0504 13:56:34.026076 (+ 23us) tablet.cc:662] Decoding operations
0504 13:56:34.026843 (+ 767us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:34.026900 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:34.026939 (+ 39us) tablet.cc:685] Acquiring locks for 5 operations
0504 13:56:34.027559 (+ 620us) tablet.cc:701] Row locks acquired
0504 13:56:34.027583 (+ 24us) write_op.cc:260] PREPARE: finished
0504 13:56:34.027741 (+ 158us) write_op.cc:270] Start()
0504 13:56:34.027848 (+ 107us) write_op.cc:276] Timestamp: P: 1777902994027663 usec, L: 0
0504 13:56:34.027875 (+ 27us) op_driver.cc:348] REPLICATION: starting
0504 13:56:34.028621 (+ 746us) log.cc:844] Serialized 2054 byte log entry
0504 13:56:34.039846 (+ 11225us) op_driver.cc:464] REPLICATION: finished
0504 13:56:34.040142 (+ 296us) write_op.cc:301] APPLY: starting
0504 13:56:34.040196 (+ 54us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:34.040330 (+ 134us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:34.040438 (+ 108us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:34.040915 (+ 477us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:34.041078 (+ 163us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0504 13:56:34.041130 (+ 52us) write_op.cc:312] APPLY: finished
0504 13:56:34.041349 (+ 219us) log.cc:844] Serialized 47 byte log entry
0504 13:56:34.041483 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:34.041560 (+ 77us) write_op.cc:454] Released schema lock
0504 13:56:34.041613 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:34.041830 (+ 217us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:34.041851 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":43245,"reported_tablets":37,"child_traces":[["op",{"apply.queue_time_us":167,"apply.run_cpu_time_us":1649,"apply.run_wall_time_us":1648,"num_ops":5,"prepare.queue_time_us":224,"prepare.run_cpu_time_us":3610,"prepare.run_wall_time_us":9971,"raft.queue_time_us":90,"raft.run_cpu_time_us":875,"raft.run_wall_time_us":875,"replication_time_us":11871}]]}
W20260504 13:56:34.317442 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 74975.5us
W20260504 13:56:34.322749 9338 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:55533 (request call id 49) took 2442 ms (client timeout 1000 ms). Trace:
W20260504 13:56:34.325150 9338 rpcz_store.cc:269] 0504 13:56:31.880261 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:31.896545 (+ 16284us) service_pool.cc:224] Handling call
0504 13:56:32.087417 (+190872us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b802470 00007fc27b8a131a 00007fc27d5b546c 00007fc27d6af3ee 00007fc27b2b64a4 00007fc27b2b6432 00007fc27b2b6361 00007fc27b2b62dc 00007fc27b2b5592 00007fc27958bbcc 00007fc27958b006
0504 13:56:34.322661 (+2235244us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25503,"reported_tablets":31,"spinlock_wait_cycles":40709888}
W20260504 13:56:34.329190 9342 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:38543 (request call id 41) took 2829 ms (client timeout 1000 ms). Trace:
I20260504 13:56:34.332190 9932 tablet_bootstrap.cc:654] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:34.329586 9342 rpcz_store.cc:269] 0504 13:56:31.499819 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:31.556319 (+ 56500us) service_pool.cc:224] Handling call
0504 13:56:34.133149 (+2576830us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:34.133173 (+ 24us) write_op.cc:454] Released schema lock
0504 13:56:34.184783 (+ 51610us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:34.184808 (+ 25us) write_op.cc:454] Released schema lock
0504 13:56:34.329134 (+144326us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:34.115390 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:34.115824 (+ 434us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:34.115849 (+ 25us) write_op.cc:435] Acquired schema lock
0504 13:56:34.115866 (+ 17us) tablet.cc:662] Decoding operations
0504 13:56:34.116135 (+ 269us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:34.116175 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:34.116202 (+ 27us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:34.116348 (+ 146us) tablet.cc:701] Row locks acquired
0504 13:56:34.116364 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:34.116474 (+ 110us) write_op.cc:270] Start()
0504 13:56:34.116555 (+ 81us) write_op.cc:276] Timestamp: P: 1777902994116400 usec, L: 0
0504 13:56:34.116587 (+ 32us) op_driver.cc:348] REPLICATION: starting
0504 13:56:34.117061 (+ 474us) log.cc:844] Serialized 533 byte log entry
0504 13:56:34.126432 (+ 9371us) op_driver.cc:464] REPLICATION: finished
0504 13:56:34.126848 (+ 416us) write_op.cc:301] APPLY: starting
0504 13:56:34.126898 (+ 50us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:34.126981 (+ 83us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:34.127055 (+ 74us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:34.127221 (+ 166us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:34.127348 (+ 127us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:34.127392 (+ 44us) write_op.cc:312] APPLY: finished
0504 13:56:34.127549 (+ 157us) log.cc:844] Serialized 23 byte log entry
0504 13:56:34.127702 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:34.127755 (+ 53us) write_op.cc:454] Released schema lock
0504 13:56:34.127787 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0504 13:56:34.160846 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:34.161365 (+ 519us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:34.161403 (+ 38us) write_op.cc:435] Acquired schema lock
0504 13:56:34.161426 (+ 23us) tablet.cc:662] Decoding operations
0504 13:56:34.161732 (+ 306us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:34.161774 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:34.161803 (+ 29us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:34.161953 (+ 150us) tablet.cc:701] Row locks acquired
0504 13:56:34.161968 (+ 15us) write_op.cc:260] PREPARE: finished
0504 13:56:34.162081 (+ 113us) write_op.cc:270] Start()
0504 13:56:34.162162 (+ 81us) write_op.cc:276] Timestamp: P: 1777902994162031 usec, L: 0
0504 13:56:34.162180 (+ 18us) op_driver.cc:348] REPLICATION: starting
0504 13:56:34.162664 (+ 484us) log.cc:844] Serialized 356 byte log entry
0504 13:56:34.173978 (+ 11314us) op_driver.cc:464] REPLICATION: finished
0504 13:56:34.174646 (+ 668us) write_op.cc:301] APPLY: starting
0504 13:56:34.174697 (+ 51us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:34.174779 (+ 82us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:34.174874 (+ 95us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:34.175025 (+ 151us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:34.175140 (+ 115us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:34.176039 (+ 899us) write_op.cc:312] APPLY: finished
0504 13:56:34.176225 (+ 186us) log.cc:844] Serialized 23 byte log entry
0504 13:56:34.176367 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:34.176441 (+ 74us) write_op.cc:454] Released schema lock
0504 13:56:34.176483 (+ 42us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":287,"apply.run_cpu_time_us":1094,"apply.run_wall_time_us":1094,"num_ops":1,"prepare.queue_time_us":129,"prepare.run_cpu_time_us":1938,"prepare.run_wall_time_us":1937,"raft.queue_time_us":1341,"raft.run_cpu_time_us":595,"raft.run_wall_time_us":596,"replication_time_us":9764}],["op",{"apply.queue_time_us":522,"apply.run_cpu_time_us":1206,"apply.run_wall_time_us":2020,"num_ops":1,"prepare.queue_time_us":668,"prepare.run_cpu_time_us":2102,"prepare.run_wall_time_us":2102,"raft.queue_time_us":3608,"raft.run_cpu_time_us":620,"raft.run_wall_time_us":620,"replication_time_us":11710}]]}
I20260504 13:56:34.235924 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e86027e3e74bc196e9a9ffc9fdf8a5" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:34.252859 9507 tablet_service.cc:1558] Processing DeleteTablet for tablet 60d3cc59ffe947fbb57b7d3301ccbac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:32 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:34.269013 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba4584755aae47e0b0f14c6c55474d18" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:34.288507 9505 tablet_service.cc:1558] Processing DeleteTablet for tablet de639fda874248f0853e74ee60aaf758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:28 UTC) from {username='slave'} at 127.0.0.1:52642
I20260504 13:56:34.290721 9517 tablet_service.cc:1558] Processing DeleteTablet for tablet bd8cd1caaeb249e3bafee7f1f0d4df2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:16 UTC) from {username='slave'} at 127.0.0.1:52642
W20260504 13:56:34.336244 9618 leader_election.cc:343] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:34.289448 11080 leader_election.cc:290] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:34.291988 9508 tablet_service.cc:1558] Processing DeleteTablet for tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 with delete_type TABLET_DATA_DELETED (replaced by tablet eef975650610423c88ce8e4cfb0b013b) from {username='slave'} at 127.0.0.1:52642
W20260504 13:56:34.291505 9767 leader_election.cc:343] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Not found: Tablet not found: 7c9030802900499a90c4dbaf5c6936a4
I20260504 13:56:34.301968 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a470ddc5bcb4121bd2f5f02f6e6b945" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:34.337378 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 45898.2us
I20260504 13:56:34.319208 11010 raft_consensus.cc:493] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:34.338344 11010 raft_consensus.cc:515] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.339349 9661 tablet_service.cc:1558] Processing DeleteTablet for tablet 60d3cc59ffe947fbb57b7d3301ccbac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:32 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:34.340152 11010 leader_election.cc:290] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:34.340695 9768 leader_election.cc:343] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 7c9030802900499a90c4dbaf5c6936a4
I20260504 13:56:34.342622 9768 leader_election.cc:304] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:34.306324 11060 consensus_queue.cc:1048] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:34.324496 10391 ts_tablet_manager.cc:1929] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:34.332230 11031 tablet.cc:1724] T bdf1c3cef3334a849f7b5e429b7f4990 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260504 13:56:34.345700 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 49357.3us
I20260504 13:56:34.346410 11010 raft_consensus.cc:2749] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:34.320348 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a470ddc5bcb4121bd2f5f02f6e6b945" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:34.323377 11091 tablet_replica.cc:333] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:34.348352 11091 raft_consensus.cc:2243] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:34.347332 9679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fae9274bb8240798f85b79504a227b3" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:34.348871 11091 raft_consensus.cc:2272] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:34.351393 11091 ts_tablet_manager.cc:1916] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:34.352769 10998 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260504 13:56:34.347856 10391 log.cc:1199] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/46fdf0ffaf454689a705949bd39deb26
I20260504 13:56:34.353992 10391 ts_tablet_manager.cc:1950] T 46fdf0ffaf454689a705949bd39deb26 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:34.355101 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4fae9274bb8240798f85b79504a227b3" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:34.355592 9679 raft_consensus.cc:2393] T 4fae9274bb8240798f85b79504a227b3 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
W20260504 13:56:34.352272 9618 leader_election.cc:343] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:34.356938 9618 leader_election.cc:343] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:34.357705 11094 tablet.cc:1724] T bdf1c3cef3334a849f7b5e429b7f4990 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:34.355511 9931 raft_consensus.cc:385] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:34.356495 9768 leader_election.cc:304] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:34.369194 11080 raft_consensus.cc:3060] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:34.375257 9931 raft_consensus.cc:740] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:34.376372 9931 consensus_queue.cc:260] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.358340 11098 tablet_replica.cc:333] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:34.384691 11098 raft_consensus.cc:2243] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:34.385267 11098 raft_consensus.cc:2272] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:34.388170 11098 ts_tablet_manager.cc:1916] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260504 13:56:34.366834 9620 leader_election.cc:343] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:34.388976 9620 leader_election.cc:304] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:34.391223 9620 leader_election.cc:343] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:34.391579 9620 leader_election.cc:304] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:34.392405 11086 raft_consensus.cc:2749] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:34.392956 11086 raft_consensus.cc:2749] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:34.394326 11090 consensus_queue.cc:1048] T fac3e36cfa4848a1ab015f89c56c2740 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:34.396100 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 257) took 2264 ms. Trace:
I20260504 13:56:34.396462 10391 rpcz_store.cc:276] 0504 13:56:32.131872 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:32.179627 (+ 47755us) service_pool.cc:224] Handling call
0504 13:56:33.865699 (+1686072us) ts_tablet_manager.cc:1151] Deleting tablet 46fdf0ffaf454689a705949bd39deb26
0504 13:56:33.865898 (+ 199us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:34.034306 (+168408us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:34.324375 (+290069us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:34.396038 (+ 71663us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14918,"tablet-delete.queue_time_us":1685260}
I20260504 13:56:34.398654 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 261) took 2265 ms. Trace:
I20260504 13:56:34.400560 9931 ts_tablet_manager.cc:1434] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.115s user 0.005s sys 0.000s
I20260504 13:56:34.405548 9932 tablet_bootstrap.cc:492] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:34.406005 9932 ts_tablet_manager.cc:1403] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.212s user 0.009s sys 0.000s
I20260504 13:56:34.408931 9932 raft_consensus.cc:359] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:34.417074 9620 leader_election.cc:343] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:34.418108 11080 raft_consensus.cc:2749] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:34.418663 9620 leader_election.cc:304] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:34.422766 11086 raft_consensus.cc:2749] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:34.431255 10391 rpcz_store.cc:276] 0504 13:56:32.133474 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:32.183796 (+ 50322us) service_pool.cc:224] Handling call
0504 13:56:34.398373 (+2214577us) ts_tablet_manager.cc:1151] Deleting tablet 930078a10afa4dd987ad021cf9609514
0504 13:56:34.398417 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:34.398602 (+ 185us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2213864}
I20260504 13:56:34.433629 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 264) took 2273 ms. Trace:
I20260504 13:56:34.434821 9931 tablet_bootstrap.cc:492] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:34.442492 9931 tablet_bootstrap.cc:654] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:34.455399 9932 raft_consensus.cc:385] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:34.435568 10013 tablet_bootstrap.cc:492] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:34.455745 9932 raft_consensus.cc:740] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:34.456193 10013 ts_tablet_manager.cc:1403] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.350s user 0.006s sys 0.003s
I20260504 13:56:34.456401 9932 consensus_queue.cc:260] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:34.458310 9932 ts_tablet_manager.cc:1434] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.052s user 0.001s sys 0.003s
I20260504 13:56:34.458906 10013 raft_consensus.cc:359] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.463295 10013 raft_consensus.cc:385] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:34.467554 10013 raft_consensus.cc:740] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:34.461367 10391 rpcz_store.cc:276] 0504 13:56:32.159957 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:32.184186 (+ 24229us) service_pool.cc:224] Handling call
0504 13:56:34.433376 (+2249190us) ts_tablet_manager.cc:1151] Deleting tablet baf07a970a7c4a35b3c849b06d6c88d0
0504 13:56:34.433415 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:34.433566 (+ 151us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2248515}
I20260504 13:56:34.477933 11098 ts_tablet_manager.cc:1929] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:34.479012 11098 log.cc:1199] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/60d3cc59ffe947fbb57b7d3301ccbac1
I20260504 13:56:34.489698 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 263) took 2330 ms. Trace:
I20260504 13:56:34.490044 10391 rpcz_store.cc:276] 0504 13:56:32.159560 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:32.185936 (+ 26376us) service_pool.cc:224] Handling call
0504 13:56:34.473056 (+2287120us) ts_tablet_manager.cc:1151] Deleting tablet 1501455a0af348e38746d100987afdd0
0504 13:56:34.473097 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:34.489634 (+ 16537us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2286507}
I20260504 13:56:34.492405 9931 tablet_bootstrap.cc:492] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:34.493515 11098 ts_tablet_manager.cc:1950] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:34.471570 10013 consensus_queue.cc:260] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.494421 9661 tablet_service.cc:1558] Processing DeleteTablet for tablet 46fdf0ffaf454689a705949bd39deb26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:28 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:34.495105 11090 raft_consensus.cc:493] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:34.495644 11090 raft_consensus.cc:515] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.496475 9931 ts_tablet_manager.cc:1403] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.002s
I20260504 13:56:34.494596 9932 tablet_bootstrap.cc:492] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:34.500288 9666 tablet_service.cc:1558] Processing DeleteTablet for tablet 8e4e583afc804212bece2f727411cc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:23 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:34.508508 10013 ts_tablet_manager.cc:1434] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.052s user 0.005s sys 0.000s
I20260504 13:56:34.509399 10013 tablet_bootstrap.cc:492] T e0b9e47b0a4443c5830422b9d0553446 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:34.528755 9660 tablet_service.cc:1558] Processing DeleteTablet for tablet 8bb77106b48a45a2aaa938dc474fa8bd with delete_type TABLET_DATA_DELETED (replaced by tablet f337e9698fe64282a684e90d9c61da5a) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:34.530824 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 265) took 2370 ms. Trace:
I20260504 13:56:34.531073 9661 tablet_service.cc:1558] Processing DeleteTablet for tablet de639fda874248f0853e74ee60aaf758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:28 UTC) from {username='slave'} at 127.0.0.1:40388
I20260504 13:56:34.531134 10391 rpcz_store.cc:276] 0504 13:56:32.160330 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:32.186301 (+ 25971us) service_pool.cc:224] Handling call
0504 13:56:34.530599 (+2344298us) ts_tablet_manager.cc:1151] Deleting tablet 604f1ba5f71041a7877daff8972a994a
0504 13:56:34.530636 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:34.530777 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2343665}
I20260504 13:56:34.532828 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 262) took 2399 ms. Trace:
I20260504 13:56:34.533114 10391 rpcz_store.cc:276] 0504 13:56:32.133844 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:32.186717 (+ 52873us) service_pool.cc:224] Handling call
0504 13:56:34.532577 (+2345860us) ts_tablet_manager.cc:1151] Deleting tablet 9ed4f5ea230a413a8dcdbb661ce173b6
0504 13:56:34.532627 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:34.532779 (+ 152us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2345256}
I20260504 13:56:34.534412 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59f9115b286d44f0b7eac7c74275cb21" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:34.534648 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 266) took 2374 ms. Trace:
W20260504 13:56:34.535573 9618 leader_election.cc:343] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:34.536062 11090 leader_election.cc:290] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:34.556999 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59f9115b286d44f0b7eac7c74275cb21" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:34.558132 9931 raft_consensus.cc:359] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.562722 9931 raft_consensus.cc:385] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:34.563050 9931 raft_consensus.cc:740] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:34.567030 9932 tablet_bootstrap.cc:654] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:34.567983 10013 tablet_bootstrap.cc:654] T e0b9e47b0a4443c5830422b9d0553446 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:34.534922 10391 rpcz_store.cc:276] 0504 13:56:32.160659 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:32.183793 (+ 23134us) service_pool.cc:224] Handling call
0504 13:56:34.534441 (+2350648us) ts_tablet_manager.cc:1151] Deleting tablet a1aba9a2a58b4e6b9e87f8ec80fcb6f1
0504 13:56:34.534476 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:34.534606 (+ 130us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2346723}
I20260504 13:56:34.570761 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 260) took 2437 ms. Trace:
I20260504 13:56:34.571133 10391 rpcz_store.cc:276] 0504 13:56:32.133075 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:32.183793 (+ 50718us) service_pool.cc:224] Handling call
0504 13:56:34.570457 (+2386664us) ts_tablet_manager.cc:1151] Deleting tablet 50e129980cd84f6ca22cbf2addb06230
0504 13:56:34.570517 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:34.570705 (+ 188us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2382367}
I20260504 13:56:34.572489 11060 raft_consensus.cc:493] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:34.572908 11060 raft_consensus.cc:515] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.574652 11060 leader_election.cc:290] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:34.576275 10391 tablet_replica.cc:333] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:34.577049 10391 raft_consensus.cc:2243] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260504 13:56:34.572682 9620 leader_election.cc:343] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:34.577535 9620 leader_election.cc:304] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:34.578174 11086 raft_consensus.cc:2749] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:34.590165 9473 leader_election.cc:343] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 8e4e583afc804212bece2f727411cc14
I20260504 13:56:34.595206 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e4e583afc804212bece2f727411cc14" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:34.595803 9828 raft_consensus.cc:2468] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:34.597086 9474 leader_election.cc:304] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:34.611722 9931 consensus_queue.cc:260] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.614740 9931 ts_tablet_manager.cc:1434] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.118s user 0.007s sys 0.000s
I20260504 13:56:34.615658 9931 tablet_bootstrap.cc:492] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:34.615605 11102 tablet.cc:1724] T bdf1c3cef3334a849f7b5e429b7f4990 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:34.629849 11091 ts_tablet_manager.cc:1929] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260504 13:56:34.630292 11091 log.cc:1199] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/60d3cc59ffe947fbb57b7d3301ccbac1
I20260504 13:56:34.632823 9932 tablet_bootstrap.cc:492] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:34.634143 9932 ts_tablet_manager.cc:1403] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.140s user 0.012s sys 0.000s
I20260504 13:56:34.636821 9932 raft_consensus.cc:359] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:34.643193 9932 raft_consensus.cc:385] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:34.647277 9932 raft_consensus.cc:740] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:34.648356 9932 consensus_queue.cc:260] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:34.650660 9932 ts_tablet_manager.cc:1434] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.016s user 0.004s sys 0.000s
I20260504 13:56:34.660118 11054 raft_consensus.cc:2804] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:34.660935 11054 raft_consensus.cc:493] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:34.661227 11054 raft_consensus.cc:3060] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:34.661974 9932 tablet_bootstrap.cc:492] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:34.669107 11054 raft_consensus.cc:515] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:34.672909 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 75947.9us
I20260504 13:56:34.675213 10391 raft_consensus.cc:2272] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:34.676880 10391 ts_tablet_manager.cc:1916] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:34.677862 9932 tablet_bootstrap.cc:654] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:34.685539 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e4e583afc804212bece2f727411cc14" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:34.686088 9832 raft_consensus.cc:3060] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:34.693938 9832 raft_consensus.cc:2468] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:34.695266 11054 leader_election.cc:290] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:34.700805 11060 consensus_queue.cc:1048] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:34.704174 11080 raft_consensus.cc:493] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:34.704607 11080 raft_consensus.cc:515] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.706389 11080 leader_election.cc:290] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:34.707325 9474 leader_election.cc:304] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:34.707957 11060 raft_consensus.cc:2804] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:34.708359 11060 raft_consensus.cc:697] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:34.709251 11060 consensus_queue.cc:237] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.631134 11091 ts_tablet_manager.cc:1950] T 60d3cc59ffe947fbb57b7d3301ccbac1 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:34.744714 10013 tablet_bootstrap.cc:492] T e0b9e47b0a4443c5830422b9d0553446 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:34.745163 10013 ts_tablet_manager.cc:1403] T e0b9e47b0a4443c5830422b9d0553446 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.236s user 0.009s sys 0.001s
I20260504 13:56:34.746786 9931 tablet_bootstrap.cc:654] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:34.749545 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0b9e47b0a4443c5830422b9d0553446" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:34.752233 10013 raft_consensus.cc:359] T e0b9e47b0a4443c5830422b9d0553446 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.755268 10013 raft_consensus.cc:385] T e0b9e47b0a4443c5830422b9d0553446 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:34.755558 10013 raft_consensus.cc:740] T e0b9e47b0a4443c5830422b9d0553446 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:34.756181 10013 consensus_queue.cc:260] T e0b9e47b0a4443c5830422b9d0553446 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.758190 10013 ts_tablet_manager.cc:1434] T e0b9e47b0a4443c5830422b9d0553446 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260504 13:56:34.762655 10013 tablet_bootstrap.cc:492] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:34.767726 10013 tablet_bootstrap.cc:654] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:34.787606 10013 tablet_bootstrap.cc:492] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:34.788086 10013 ts_tablet_manager.cc:1403] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.000s
I20260504 13:56:34.790294 10013 raft_consensus.cc:359] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.791025 10013 raft_consensus.cc:385] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:34.791337 10013 raft_consensus.cc:740] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:34.795595 10013 consensus_queue.cc:260] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.798200 10013 ts_tablet_manager.cc:1434] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20260504 13:56:34.799140 10013 tablet_bootstrap.cc:492] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:34.805632 9767 leader_election.cc:343] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:34.832114 11091 tablet_replica.cc:333] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:34.832989 11091 raft_consensus.cc:2243] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:34.838030 10013 tablet_bootstrap.cc:654] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:34.839848 9931 tablet_bootstrap.cc:492] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:34.848757 9931 ts_tablet_manager.cc:1403] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.233s user 0.010s sys 0.000s
I20260504 13:56:34.849176 11091 raft_consensus.cc:2272] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:34.861400 11091 ts_tablet_manager.cc:1916] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:34.872690 9931 raft_consensus.cc:359] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:34.877684 9931 raft_consensus.cc:385] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:34.878298 9931 raft_consensus.cc:740] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:34.879607 9931 consensus_queue.cc:260] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:34.889030 10013 tablet_bootstrap.cc:492] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:34.889456 10013 ts_tablet_manager.cc:1403] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.090s user 0.005s sys 0.004s
I20260504 13:56:34.893517 9931 ts_tablet_manager.cc:1434] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.044s user 0.004s sys 0.000s
I20260504 13:56:34.897730 9931 tablet_bootstrap.cc:492] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:34.905340 11111 log.cc:927] Time spent T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7: Append to log took a long time: real 0.063s user 0.031s sys 0.000s
I20260504 13:56:34.923761 10013 raft_consensus.cc:359] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.924397 10013 raft_consensus.cc:385] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:34.924638 10013 raft_consensus.cc:740] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:34.925212 10013 consensus_queue.cc:260] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.927315 10013 ts_tablet_manager.cc:1434] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20260504 13:56:34.928120 10013 tablet_bootstrap.cc:492] T c47b6e3b4d0c45c28384f7aae6b749de P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:34.947204 9679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0b9e47b0a4443c5830422b9d0553446" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:34.944396 9931 tablet_bootstrap.cc:654] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:34.948637 9679 raft_consensus.cc:2468] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:34.951540 10681 raft_consensus.cc:493] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:34.952243 10681 raft_consensus.cc:515] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.952876 11086 raft_consensus.cc:493] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:34.953464 11086 raft_consensus.cc:515] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:34.954538 10681 leader_election.cc:290] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:34.960959 11010 raft_consensus.cc:493] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:34.961419 11010 raft_consensus.cc:515] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:34.966140 11086 leader_election.cc:290] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:34.971611 10391 ts_tablet_manager.cc:1929] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:34.972003 10391 log.cc:1199] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/8bb77106b48a45a2aaa938dc474fa8bd
I20260504 13:56:34.972592 10391 ts_tablet_manager.cc:1950] T 8bb77106b48a45a2aaa938dc474fa8bd P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:34.972909 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb77106b48a45a2aaa938dc474fa8bd" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:34.976212 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de6714ee86d64112baf13dd1b2b87582" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:34.976724 10013 tablet_bootstrap.cc:654] T c47b6e3b4d0c45c28384f7aae6b749de P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:34.977774 9767 leader_election.cc:343] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: Tablet not RUNNING: SHUTDOWN
I20260504 13:56:34.977736 9768 leader_election.cc:304] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:34.978727 11080 raft_consensus.cc:2804] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:34.980403 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1a01231bb044433ae5827d07a6592f8" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:34.979076 11080 raft_consensus.cc:493] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:34.983304 11010 leader_election.cc:290] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:34.983421 11080 raft_consensus.cc:3060] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:34.984115 9618 leader_election.cc:343] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:34.984833 9618 leader_election.cc:343] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:34.999329 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1a01231bb044433ae5827d07a6592f8" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:35.000689 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de6714ee86d64112baf13dd1b2b87582" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:35.001636 9473 leader_election.cc:343] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 8e4e583afc804212bece2f727411cc14
I20260504 13:56:35.003588 10956 consensus_queue.cc:1048] T 4fae9274bb8240798f85b79504a227b3 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:34.954980 10681 raft_consensus.cc:493] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:35.008337 10681 raft_consensus.cc:515] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:35.010092 9932 tablet_bootstrap.cc:492] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:35.010360 10681 leader_election.cc:290] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:35.010648 9932 ts_tablet_manager.cc:1403] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.349s user 0.015s sys 0.008s
I20260504 13:56:35.011861 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88619eaa917d4c33a910fc1347ab126e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:35.013284 9618 leader_election.cc:343] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.017428 11091 ts_tablet_manager.cc:1929] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:35.017833 11091 log.cc:1199] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/7c9030802900499a90c4dbaf5c6936a4
I20260504 13:56:35.018462 11091 ts_tablet_manager.cc:1950] T 7c9030802900499a90c4dbaf5c6936a4 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:35.051488 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88619eaa917d4c33a910fc1347ab126e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:35.062841 10013 tablet_bootstrap.cc:492] T c47b6e3b4d0c45c28384f7aae6b749de P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:35.070747 9931 tablet_bootstrap.cc:492] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:35.014624 11080 raft_consensus.cc:515] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.071421 9931 ts_tablet_manager.cc:1403] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.174s user 0.023s sys 0.002s
I20260504 13:56:35.074611 9931 raft_consensus.cc:359] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.075563 9931 raft_consensus.cc:385] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:35.076268 10013 ts_tablet_manager.cc:1403] T c47b6e3b4d0c45c28384f7aae6b749de P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.148s user 0.010s sys 0.000s
I20260504 13:56:35.084502 9931 raft_consensus.cc:740] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:35.085470 9931 consensus_queue.cc:260] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.091189 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0b9e47b0a4443c5830422b9d0553446" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:35.091727 9534 raft_consensus.cc:3060] T e0b9e47b0a4443c5830422b9d0553446 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:35.093014 9620 leader_election.cc:343] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.093451 9620 leader_election.cc:304] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:35.095384 9932 raft_consensus.cc:359] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:35.096640 9932 raft_consensus.cc:385] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:35.107975 9534 raft_consensus.cc:2468] T e0b9e47b0a4443c5830422b9d0553446 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:35.087594 10013 raft_consensus.cc:359] T c47b6e3b4d0c45c28384f7aae6b749de P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.109345 10013 raft_consensus.cc:385] T c47b6e3b4d0c45c28384f7aae6b749de P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:35.110744 10013 raft_consensus.cc:740] T c47b6e3b4d0c45c28384f7aae6b749de P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:35.119779 10013 consensus_queue.cc:260] T c47b6e3b4d0c45c28384f7aae6b749de P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.090710 11080 leader_election.cc:290] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:35.128500 9328 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 604f1ba5f71041a7877daff8972a994a (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 604f1ba5f71041a7877daff8972a994a consensus state has no leader
I20260504 13:56:35.131453 10013 ts_tablet_manager.cc:1434] T c47b6e3b4d0c45c28384f7aae6b749de P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.055s user 0.007s sys 0.000s
I20260504 13:56:35.132654 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 269) took 2971 ms. Trace:
I20260504 13:56:35.134341 10391 rpcz_store.cc:276] 0504 13:56:32.191439 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:32.195254 (+ 3815us) service_pool.cc:224] Handling call
0504 13:56:34.576143 (+2380889us) ts_tablet_manager.cc:1151] Deleting tablet 8bb77106b48a45a2aaa938dc474fa8bd
0504 13:56:34.576180 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:34.879775 (+303595us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:34.971467 (+ 91692us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:35.132597 (+161130us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2380385}
I20260504 13:56:35.136113 10391 tablet_replica.cc:333] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:35.137077 10391 raft_consensus.cc:2243] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
W20260504 13:56:35.139390 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 151360us
I20260504 13:56:35.147244 10391 raft_consensus.cc:2272] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:35.147836 11086 raft_consensus.cc:2749] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:35.152699 10391 ts_tablet_manager.cc:1916] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:35.165439 10013 tablet_bootstrap.cc:492] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:35.148463 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 55599.1us
I20260504 13:56:35.175652 9677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0b9e47b0a4443c5830422b9d0553446" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
W20260504 13:56:35.176587 9620 leader_election.cc:343] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.177006 9620 leader_election.cc:304] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:35.176534 9677 raft_consensus.cc:3060] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:35.178020 11086 raft_consensus.cc:2749] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:35.151348 9932 raft_consensus.cc:740] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
W20260504 13:56:35.181485 9620 leader_election.cc:343] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.181957 9620 leader_election.cc:304] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:35.183240 11086 raft_consensus.cc:2749] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:35.204213 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 101820us
I20260504 13:56:35.205262 9767 leader_election.cc:304] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:35.205904 11010 raft_consensus.cc:2804] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:35.206373 11010 raft_consensus.cc:697] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:35.207286 11010 consensus_queue.cc:237] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.179159 9932 consensus_queue.cc:260] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:35.216660 9932 ts_tablet_manager.cc:1434] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.206s user 0.009s sys 0.000s
I20260504 13:56:35.218770 9932 tablet_bootstrap.cc:492] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:35.230468 9932 tablet_bootstrap.cc:654] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:35.233294 9931 ts_tablet_manager.cc:1434] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.161s user 0.005s sys 0.001s
I20260504 13:56:35.234097 9931 tablet_bootstrap.cc:492] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:35.212780 11012 raft_consensus.cc:493] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:35.236763 11012 raft_consensus.cc:515] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.239342 11012 leader_election.cc:290] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:35.242662 9677 raft_consensus.cc:2468] T e0b9e47b0a4443c5830422b9d0553446 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:35.247488 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f6b3e969760442283c3191e062c1839" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:35.249984 9679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f6b3e969760442283c3191e062c1839" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:35.250662 9679 raft_consensus.cc:2468] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
W20260504 13:56:35.253901 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 48698us
W20260504 13:56:35.269975 9474 leader_election.cc:343] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.273234 9932 tablet_bootstrap.cc:492] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:35.273633 9932 ts_tablet_manager.cc:1403] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.003s
I20260504 13:56:35.281951 9932 raft_consensus.cc:359] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.282509 9932 raft_consensus.cc:385] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:35.282781 9932 raft_consensus.cc:740] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:35.284576 9932 consensus_queue.cc:260] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:35.316727 9768 leader_election.cc:343] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 8bb77106b48a45a2aaa938dc474fa8bd
I20260504 13:56:35.317239 9768 leader_election.cc:304] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:35.321870 11010 raft_consensus.cc:2749] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:35.323505 9931 tablet_bootstrap.cc:654] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:35.325695 11026 consensus_queue.cc:1048] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:35.334160 11063 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:35.334892 11063 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
W20260504 13:56:35.337848 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 84162.4us
I20260504 13:56:35.344887 10013 tablet_bootstrap.cc:654] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:35.344722 10768 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:53088 (request call id 31) took 1371 ms. Trace:
I20260504 13:56:35.345184 10768 rpcz_store.cc:276] 0504 13:56:33.973028 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:33.973157 (+ 129us) service_pool.cc:224] Handling call
0504 13:56:35.343625 (+1370468us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54000104d4. stack: 00007f4beb944c83 00007f4beb944b80 00007f4beb4b5574 000000000050e094 000000000050e029 00007f4bf299d470 00007f4bf2a3c31a 00007f4bf475046c 00007f4bf484d6a9 00007f4bf2453a24 00007f4bf24539b2 00007f4bf24538e1 00007f4bf245385c 00007f4bf2452b12 00007f4bf0726bcc 00007f4bf0726006
0504 13:56:35.344520 (+ 895us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3137950592}
I20260504 13:56:35.325350 9932 ts_tablet_manager.cc:1434] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.046s user 0.003s sys 0.002s
I20260504 13:56:35.350078 9473 leader_election.cc:304] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:35.352214 10956 raft_consensus.cc:2804] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:35.352531 10956 raft_consensus.cc:493] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:35.352828 10956 raft_consensus.cc:3060] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:35.355979 9932 tablet_bootstrap.cc:492] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:35.360235 10956 raft_consensus.cc:515] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.362414 10956 leader_election.cc:290] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:35.362679 9932 tablet_bootstrap.cc:654] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:35.364672 11128 consensus_queue.cc:1048] T 4cf1504f0202481ba8b08f9bec5bc3d9 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:35.369760 9677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f6b3e969760442283c3191e062c1839" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:35.370334 9677 raft_consensus.cc:3060] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:35.377367 9677 raft_consensus.cc:2468] T 2f6b3e969760442283c3191e062c1839 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:35.383586 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 4 candidate_status { last_received { term: 3 index: 351 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:35.386798 11063 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
W20260504 13:56:35.354802 10791 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:50343 (request call id 21) took 1410 ms (client timeout 1000 ms). Trace:
I20260504 13:56:35.388201 9473 leader_election.cc:304] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
W20260504 13:56:35.387926 10791 rpcz_store.cc:269] 0504 13:56:34.037525 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:34.037635 (+ 110us) service_pool.cc:224] Handling call
0504 13:56:35.351732 (+1314097us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54000104d4. stack: 00007f4beb944c83 00007f4beb944b80 00007f4beb4b5574 000000000050e094 000000000050e029 00007f4bf299d470 00007f4bf2a3c31a 00007f4bf475046c 00007f4bf484a3ee 00007f4bf24514a4 00007f4bf2451432 00007f4bf2451361 00007f4bf24512dc 00007f4bf2450592 00007f4bf0726bcc 00007f4bf0726006
0504 13:56:35.354649 (+ 2917us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2987200384}
I20260504 13:56:35.369377 11074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec9f4c3a43d84fc9a420dd10723c01a1. Latest consensus state: current_term: 3 leader_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:35.390043 11074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:56:35.395785 11128 raft_consensus.cc:2804] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:35.396835 11128 raft_consensus.cc:697] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:35.397948 10013 tablet_bootstrap.cc:492] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:35.397570 11128 consensus_queue.cc:237] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.398315 10013 ts_tablet_manager.cc:1403] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.233s user 0.010s sys 0.000s
W20260504 13:56:35.401237 10776 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:53697 (request call id 9) took 1401 ms (client timeout 1000 ms). Trace:
W20260504 13:56:35.402078 10776 rpcz_store.cc:269] 0504 13:56:33.949524 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:33.960926 (+ 11402us) service_pool.cc:224] Handling call
0504 13:56:35.347275 (+1386349us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54000104d4. stack: 00007f4beb944c83 00007f4beb944b80 00007f4beb4b5574 000000000050e094 000000000050e029 00007f4bf299d470 00007f4bf2a3c31a 00007f4bf475046c 00007f4bf484a3ee 00007f4bf24514a4 00007f4bf2451432 00007f4bf2451361 00007f4bf24512dc 00007f4bf2450592 00007f4bf0726bcc 00007f4bf0726006
0504 13:56:35.351086 (+ 3811us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3157002112}
W20260504 13:56:35.373248 10777 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:56157 (request call id 23) took 1330 ms (client timeout 1000 ms). Trace:
W20260504 13:56:35.404311 10777 rpcz_store.cc:269] 0504 13:56:34.043067 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:34.043166 (+ 99us) service_pool.cc:224] Handling call
0504 13:56:35.359713 (+1316547us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54000104d4. stack: 00007f4beb944c83 00007f4beb944b80 00007f4beb4b5574 000000000050e094 000000000050e029 00007f4bf299d470 00007f4bf2a3c31a 00007f4bf475046c 00007f4bf484a3ee 00007f4bf24514a4 00007f4bf2451432 00007f4bf2451361 00007f4bf24512dc 00007f4bf2450592 00007f4bf0726bcc 00007f4bf0726006
0504 13:56:35.373055 (+ 13342us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2973680000}
I20260504 13:56:35.406209 10013 raft_consensus.cc:359] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.414736 9826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f6b3e969760442283c3191e062c1839" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:35.415268 10013 raft_consensus.cc:385] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:35.415517 10013 raft_consensus.cc:740] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:35.416790 10013 consensus_queue.cc:260] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.418160 10391 ts_tablet_manager.cc:1929] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260504 13:56:35.421937 9474 leader_election.cc:343] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.425354 10391 log.cc:1199] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/60d3cc59ffe947fbb57b7d3301ccbac1
I20260504 13:56:35.426344 10391 ts_tablet_manager.cc:1950] T 60d3cc59ffe947fbb57b7d3301ccbac1 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:35.428478 9931 tablet_bootstrap.cc:492] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:35.428988 9931 ts_tablet_manager.cc:1403] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.195s user 0.010s sys 0.000s
I20260504 13:56:35.436203 9932 tablet_bootstrap.cc:492] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:35.436720 9932 ts_tablet_manager.cc:1403] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.081s user 0.006s sys 0.004s
I20260504 13:56:35.439244 9932 raft_consensus.cc:359] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.440040 9932 raft_consensus.cc:385] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:35.440330 9932 raft_consensus.cc:740] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:35.441016 9932 consensus_queue.cc:260] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.443136 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 271) took 1190 ms. Trace:
I20260504 13:56:35.445150 10391 rpcz_store.cc:276] 0504 13:56:34.252622 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:34.252736 (+ 114us) service_pool.cc:224] Handling call
0504 13:56:35.135940 (+883204us) ts_tablet_manager.cc:1151] Deleting tablet 60d3cc59ffe947fbb57b7d3301ccbac1
0504 13:56:35.135985 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:35.299138 (+163153us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:35.418033 (+118895us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:35.443066 (+ 25033us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8617,"tablet-delete.queue_time_us":799935}
I20260504 13:56:35.449033 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 273) took 1160 ms. Trace:
I20260504 13:56:35.449426 10391 rpcz_store.cc:276] 0504 13:56:34.288268 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:34.288379 (+ 111us) service_pool.cc:224] Handling call
0504 13:56:35.448740 (+1160361us) ts_tablet_manager.cc:1151] Deleting tablet de639fda874248f0853e74ee60aaf758
0504 13:56:35.448790 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:35.448976 (+ 186us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1112097}
I20260504 13:56:35.452448 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 274) took 1162 ms. Trace:
I20260504 13:56:35.453065 10391 rpcz_store.cc:276] 0504 13:56:34.290250 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:34.290614 (+ 364us) service_pool.cc:224] Handling call
0504 13:56:35.452168 (+1161554us) ts_tablet_manager.cc:1151] Deleting tablet bd8cd1caaeb249e3bafee7f1f0d4df2f
0504 13:56:35.452216 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:35.452393 (+ 177us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1115340}
I20260504 13:56:35.454698 10391 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52642 (request call id 275) took 1163 ms. Trace:
I20260504 13:56:35.455117 10391 rpcz_store.cc:276] 0504 13:56:34.291763 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:34.291879 (+ 116us) service_pool.cc:224] Handling call
0504 13:56:35.454411 (+1162532us) ts_tablet_manager.cc:1151] Deleting tablet 41d41f8ba6dc4bb18116cb2b2dfabff0
0504 13:56:35.454457 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:35.454646 (+ 189us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1117040}
I20260504 13:56:35.457893 9932 ts_tablet_manager.cc:1434] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.021s user 0.003s sys 0.002s
I20260504 13:56:35.460757 9932 tablet_bootstrap.cc:492] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:35.468544 10814 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53064 (request call id 4) took 1714 ms. Trace:
I20260504 13:56:35.470749 10013 ts_tablet_manager.cc:1434] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.072s user 0.005s sys 0.004s
I20260504 13:56:35.471745 10013 tablet_bootstrap.cc:492] T de6714ee86d64112baf13dd1b2b87582 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:35.474673 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 45833.2us
I20260504 13:56:35.503952 11139 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
W20260504 13:56:35.508587 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 68303.8us
I20260504 13:56:35.509109 9931 raft_consensus.cc:359] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.509735 9931 raft_consensus.cc:385] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:35.509996 9931 raft_consensus.cc:740] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:35.510566 9931 consensus_queue.cc:260] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.510197 11139 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:35.512501 9931 ts_tablet_manager.cc:1434] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.083s user 0.005s sys 0.000s
I20260504 13:56:35.513309 9931 tablet_bootstrap.cc:492] T a26c6a2ffb8746a3941a56c7259412e3 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:35.515588 9932 tablet_bootstrap.cc:654] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:35.519711 10810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 4 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:35.520354 10810 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 806407748d564439807fe5d73db9abda in term 3.
I20260504 13:56:35.521341 9931 tablet_bootstrap.cc:654] T a26c6a2ffb8746a3941a56c7259412e3 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:35.521668 9173 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda; no voters:
I20260504 13:56:35.524036 11125 raft_consensus.cc:493] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:35.524536 11125 raft_consensus.cc:515] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.528561 11125 leader_election.cc:290] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:35.529286 9679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a1a77ad56634e068401a2bfbc977761" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:35.529871 9679 raft_consensus.cc:2468] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:35.531054 9768 leader_election.cc:304] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:35.531965 11125 raft_consensus.cc:2804] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:35.531386 11139 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:35.532402 11125 raft_consensus.cc:493] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:35.532815 11125 raft_consensus.cc:3060] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:35.534309 11139 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260504 13:56:35.540303 11125 raft_consensus.cc:515] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.538051 11139 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 3 FOLLOWER]: Starting leader election (detected failure of leader ec9f4c3a43d84fc9a420dd10723c01a1)
I20260504 13:56:35.542572 11139 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 3 FOLLOWER]: Advancing to term 4
I20260504 13:56:35.580294 9932 tablet_bootstrap.cc:492] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:35.580798 9932 ts_tablet_manager.cc:1403] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.120s user 0.004s sys 0.006s
I20260504 13:56:35.583150 9932 raft_consensus.cc:359] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.583921 9932 raft_consensus.cc:385] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:35.584821 9932 raft_consensus.cc:740] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:35.593765 10681 raft_consensus.cc:493] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:35.594192 10681 raft_consensus.cc:515] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.595888 10681 leader_election.cc:290] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:35.597606 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d27b04d3f4443c5ade9616d8626d50e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:35.598866 9931 tablet_bootstrap.cc:492] T a26c6a2ffb8746a3941a56c7259412e3 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:35.599292 9931 ts_tablet_manager.cc:1403] T a26c6a2ffb8746a3941a56c7259412e3 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.086s user 0.009s sys 0.000s
W20260504 13:56:35.599294 9620 leader_election.cc:343] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.593127 9932 consensus_queue.cc:260] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.614753 9679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a1a77ad56634e068401a2bfbc977761" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:35.615372 9679 raft_consensus.cc:3060] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:35.622593 9679 raft_consensus.cc:2468] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:35.623651 9768 leader_election.cc:304] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:35.624393 11010 raft_consensus.cc:2804] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:35.624814 11010 raft_consensus.cc:697] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:35.625555 11010 consensus_queue.cc:237] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.636000 9932 ts_tablet_manager.cc:1434] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.050s user 0.006s sys 0.002s
I20260504 13:56:35.641417 9932 tablet_bootstrap.cc:492] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:35.651583 11125 leader_election.cc:290] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:35.679631 11145 raft_consensus.cc:493] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 13:56:35.679936 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 101480us
I20260504 13:56:35.680413 11145 raft_consensus.cc:515] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.657015 11139 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:35.682360 9198 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 4 FOLLOWER]: Rejecting Update request from peer ec9f4c3a43d84fc9a420dd10723c01a1 for earlier term 3. Current term is 4. Ops: []
I20260504 13:56:35.682770 10956 raft_consensus.cc:493] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:35.683145 10956 raft_consensus.cc:515] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.684212 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a1a77ad56634e068401a2bfbc977761" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:35.684828 10956 leader_election.cc:290] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:35.680708 9931 raft_consensus.cc:359] T a26c6a2ffb8746a3941a56c7259412e3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.687644 9931 raft_consensus.cc:385] T a26c6a2ffb8746a3941a56c7259412e3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:35.687912 9931 raft_consensus.cc:740] T a26c6a2ffb8746a3941a56c7259412e3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:35.690449 11139 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 4 election: Requested vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:35.691439 9931 consensus_queue.cc:260] T a26c6a2ffb8746a3941a56c7259412e3 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.693471 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a26c6a2ffb8746a3941a56c7259412e3" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:35.694007 9681 raft_consensus.cc:2468] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:35.696166 10810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 4 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7"
I20260504 13:56:35.696660 10810 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 3 FOLLOWER]: Advancing to term 4
I20260504 13:56:35.696417 11142 consensus_queue.cc:1059] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 }, Status: INVALID_TERM, Last received: 3.354, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20260504 13:56:35.708938 10810 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 806407748d564439807fe5d73db9abda in term 4.
I20260504 13:56:35.710297 9931 ts_tablet_manager.cc:1434] T a26c6a2ffb8746a3941a56c7259412e3 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.111s user 0.005s sys 0.000s
I20260504 13:56:35.711179 9931 tablet_bootstrap.cc:492] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:35.710520 9932 tablet_bootstrap.cc:654] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:35.718572 9173 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda; no voters:
I20260504 13:56:35.719455 11139 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 4 FOLLOWER]: Leader election won for term 4
I20260504 13:56:35.721024 11139 raft_consensus.cc:697] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 4 LEADER]: Becoming Leader. State: Replica: 806407748d564439807fe5d73db9abda, State: Running, Role: LEADER
I20260504 13:56:35.721935 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2bfb681e6824e9f9bdcbea23061d57a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:35.723218 9620 leader_election.cc:343] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.723556 11139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 354, Committed index: 354, Last appended: 3.354, Last appended by leader: 354, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:35.723446 11145 leader_election.cc:290] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:35.726568 11010 raft_consensus.cc:493] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:35.727027 11010 raft_consensus.cc:515] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.728875 11010 leader_election.cc:290] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:35.731581 9679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47b6e3b4d0c45c28384f7aae6b749de" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:35.732182 9679 raft_consensus.cc:2468] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:35.733026 11145 raft_consensus.cc:493] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:35.733378 9768 leader_election.cc:304] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:35.733440 11145 raft_consensus.cc:515] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.734197 11010 raft_consensus.cc:2804] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:35.734567 11010 raft_consensus.cc:493] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:35.734963 11010 raft_consensus.cc:3060] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:35.735688 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a629c2a69084b6b98984c8d374eeb1e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:35.736989 9620 leader_election.cc:343] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.737488 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a1a77ad56634e068401a2bfbc977761" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:35.742017 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d27b04d3f4443c5ade9616d8626d50e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:35.743045 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2bfb681e6824e9f9bdcbea23061d57a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:35.746266 9618 leader_election.cc:343] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.746658 9618 leader_election.cc:304] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:35.746829 9767 leader_election.cc:343] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:35.747651 9618 leader_election.cc:343] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:35.747809 9767 leader_election.cc:343] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.747820 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47b6e3b4d0c45c28384f7aae6b749de" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:35.747977 9618 leader_election.cc:304] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:35.748333 9535 raft_consensus.cc:2468] T c47b6e3b4d0c45c28384f7aae6b749de P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:35.749148 10681 raft_consensus.cc:2749] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:35.749617 10681 raft_consensus.cc:2749] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:35.749941 9473 leader_election.cc:304] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:35.751365 11135 raft_consensus.cc:2804] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:35.751667 11135 raft_consensus.cc:493] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:35.751962 11135 raft_consensus.cc:3060] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:35.751135 9826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a26c6a2ffb8746a3941a56c7259412e3" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:35.753347 9826 raft_consensus.cc:2468] T a26c6a2ffb8746a3941a56c7259412e3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:35.764694 11145 leader_election.cc:290] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
W20260504 13:56:35.769384 9473 consensus_peers.cc:597] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e4e583afc804212bece2f727411cc14. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:35.767441 11145 raft_consensus.cc:493] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:35.768950 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a629c2a69084b6b98984c8d374eeb1e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:35.772780 9618 leader_election.cc:343] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.772675 11010 raft_consensus.cc:515] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.773281 9618 leader_election.cc:304] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:35.774185 11086 raft_consensus.cc:2749] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:35.774232 10956 consensus_queue.cc:1048] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:35.775084 11060 raft_consensus.cc:493] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:35.775394 11010 leader_election.cc:290] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:35.775578 9679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47b6e3b4d0c45c28384f7aae6b749de" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:35.775516 11060 raft_consensus.cc:515] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.777204 11060 leader_election.cc:290] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:35.776113 9679 raft_consensus.cc:3060] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:35.775993 11145 raft_consensus.cc:515] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.781944 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47b6e3b4d0c45c28384f7aae6b749de" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:35.782459 9535 raft_consensus.cc:3060] T c47b6e3b4d0c45c28384f7aae6b749de P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:35.782603 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e86027e3e74bc196e9a9ffc9fdf8a5" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:35.786517 10013 tablet_bootstrap.cc:654] T de6714ee86d64112baf13dd1b2b87582 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:35.789997 9474 leader_election.cc:343] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.791510 9931 tablet_bootstrap.cc:654] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:35.795099 9535 raft_consensus.cc:2468] T c47b6e3b4d0c45c28384f7aae6b749de P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:35.799873 9767 leader_election.cc:304] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:35.758734 11135 raft_consensus.cc:515] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.800588 11125 raft_consensus.cc:2804] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:35.801039 11125 raft_consensus.cc:697] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:35.801821 11125 consensus_queue.cc:237] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.804124 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e86027e3e74bc196e9a9ffc9fdf8a5" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:35.804697 9683 raft_consensus.cc:2468] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:35.804826 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d9bd2722d23420ea0eac7664357a60c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:35.805548 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a26c6a2ffb8746a3941a56c7259412e3" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:35.805804 9618 leader_election.cc:343] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.806057 9833 raft_consensus.cc:3060] T a26c6a2ffb8746a3941a56c7259412e3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:35.805855 11135 leader_election.cc:290] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:35.809779 11145 leader_election.cc:290] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:35.824932 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d9bd2722d23420ea0eac7664357a60c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:35.832633 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a26c6a2ffb8746a3941a56c7259412e3" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:35.833112 9681 raft_consensus.cc:3060] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:35.835776 9931 tablet_bootstrap.cc:492] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:35.836146 9931 ts_tablet_manager.cc:1403] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.125s user 0.009s sys 0.000s
W20260504 13:56:35.839996 9620 leader_election.cc:343] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.840366 9620 leader_election.cc:304] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:35.840626 9681 raft_consensus.cc:2468] T a26c6a2ffb8746a3941a56c7259412e3 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:35.841060 11145 raft_consensus.cc:2749] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:35.828589 9473 leader_election.cc:304] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:35.845993 11026 raft_consensus.cc:2804] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:35.846347 11026 raft_consensus.cc:493] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:35.846887 11026 raft_consensus.cc:3060] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:35.848311 9833 raft_consensus.cc:2468] T a26c6a2ffb8746a3941a56c7259412e3 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:35.852152 9474 leader_election.cc:304] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:35.852783 11060 raft_consensus.cc:2804] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:35.853183 11060 raft_consensus.cc:697] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:35.853912 11060 consensus_queue.cc:237] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.861403 11140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 806407748d564439807fe5d73db9abda. Latest consensus state: current_term: 4 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:35.862146 11140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: This master's current role is: LEADER
I20260504 13:56:35.859114 11026 raft_consensus.cc:515] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.891304 11026 leader_election.cc:290] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:35.892978 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e86027e3e74bc196e9a9ffc9fdf8a5" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:35.893466 9683 raft_consensus.cc:3060] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:35.901297 11146 raft_consensus.cc:3055] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 3 LEADER]: Stepping down as leader of term 3
I20260504 13:56:35.901636 11146 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ec9f4c3a43d84fc9a420dd10723c01a1, State: Running, Role: LEADER
I20260504 13:56:35.902345 11146 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 354, Committed index: 354, Last appended: 3.354, Last appended by leader: 354, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:35.906805 11155 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260504 13:56:35.913475 10013 tablet_bootstrap.cc:492] T de6714ee86d64112baf13dd1b2b87582 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:35.913903 10013 ts_tablet_manager.cc:1403] T de6714ee86d64112baf13dd1b2b87582 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.442s user 0.005s sys 0.006s
I20260504 13:56:35.894800 9679 raft_consensus.cc:2468] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:35.900458 9683 raft_consensus.cc:2468] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:35.905128 9932 tablet_bootstrap.cc:492] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:35.915125 9932 ts_tablet_manager.cc:1403] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.275s user 0.015s sys 0.005s
I20260504 13:56:35.915830 9473 leader_election.cc:304] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:35.916296 10013 raft_consensus.cc:359] T de6714ee86d64112baf13dd1b2b87582 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.916874 10013 raft_consensus.cc:385] T de6714ee86d64112baf13dd1b2b87582 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:35.917128 10013 raft_consensus.cc:740] T de6714ee86d64112baf13dd1b2b87582 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:35.917378 9932 raft_consensus.cc:359] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:35.917976 9932 raft_consensus.cc:385] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:35.917732 10013 consensus_queue.cc:260] T de6714ee86d64112baf13dd1b2b87582 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.918437 11135 raft_consensus.cc:2804] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:35.919234 11145 raft_consensus.cc:493] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:35.919651 10013 ts_tablet_manager.cc:1434] T de6714ee86d64112baf13dd1b2b87582 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260504 13:56:35.919653 11145 raft_consensus.cc:515] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:35.920477 10013 tablet_bootstrap.cc:492] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:35.921319 11145 leader_election.cc:290] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:35.918215 9932 raft_consensus.cc:740] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:35.923309 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a862a43c5ac443d9bac98e0472ca23b2" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:35.924321 9618 leader_election.cc:343] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.923565 9932 consensus_queue.cc:260] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:35.925729 9932 ts_tablet_manager.cc:1434] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
W20260504 13:56:35.928326 11153 log.cc:927] Time spent T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260504 13:56:35.918983 11135 raft_consensus.cc:697] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:35.932047 11135 consensus_queue.cc:237] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.936035 9826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e86027e3e74bc196e9a9ffc9fdf8a5" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:35.937156 9474 leader_election.cc:343] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.951690 11145 raft_consensus.cc:493] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:35.952171 11145 raft_consensus.cc:515] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:35.954718 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c13c4cd2f66426f8e7e81eeac7bc0f3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:35.955935 9618 leader_election.cc:343] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.955443 11145 leader_election.cc:290] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:35.963608 11146 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 3 FOLLOWER]: Advancing to term 4
I20260504 13:56:35.965179 9932 tablet_bootstrap.cc:492] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:35.972220 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c13c4cd2f66426f8e7e81eeac7bc0f3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:35.973358 9620 leader_election.cc:343] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.973727 9620 leader_election.cc:304] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:35.974977 11145 raft_consensus.cc:2749] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:35.976852 9931 raft_consensus.cc:359] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.977543 9931 raft_consensus.cc:385] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:35.977797 9931 raft_consensus.cc:740] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:35.982044 9931 consensus_queue.cc:260] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:35.989084 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a862a43c5ac443d9bac98e0472ca23b2" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:35.990190 9620 leader_election.cc:343] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:35.990566 9620 leader_election.cc:304] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:35.991453 11145 raft_consensus.cc:2749] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:35.997623 9931 ts_tablet_manager.cc:1434] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.161s user 0.007s sys 0.000s
I20260504 13:56:36.006266 10013 tablet_bootstrap.cc:654] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:36.025908 9931 tablet_bootstrap.cc:492] T 2f6b3e969760442283c3191e062c1839 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:36.026268 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:36.031844 11145 raft_consensus.cc:493] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:36.032370 11145 raft_consensus.cc:515] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.034641 11145 leader_election.cc:290] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:36.035492 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b84c3bedc7a84e0d8a873fb15689b314" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:36.035670 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b84c3bedc7a84e0d8a873fb15689b314" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:36.036670 9932 tablet_bootstrap.cc:654] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:36.039661 9620 leader_election.cc:343] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:36.038515 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 53312.9us
W20260504 13:56:36.063832 9176 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.238.252:44781 timed out after 0.533s (SENT)
W20260504 13:56:36.071905 9618 leader_election.cc:343] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:36.072258 9618 leader_election.cc:304] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:36.077278 11145 raft_consensus.cc:2749] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:36.086490 11125 consensus_queue.cc:1048] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:36.103888 9931 tablet_bootstrap.cc:654] T 2f6b3e969760442283c3191e062c1839 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:36.132393 9932 tablet_bootstrap.cc:492] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:36.132803 9932 ts_tablet_manager.cc:1403] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.176s user 0.009s sys 0.000s
I20260504 13:56:36.150763 9932 raft_consensus.cc:359] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.151415 9932 raft_consensus.cc:385] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:36.151670 9932 raft_consensus.cc:740] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:36.165325 10013 tablet_bootstrap.cc:492] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:36.165274 11010 consensus_queue.cc:1048] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:36.165705 10013 ts_tablet_manager.cc:1403] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.245s user 0.009s sys 0.007s
I20260504 13:56:36.169019 10013 raft_consensus.cc:359] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.169641 10013 raft_consensus.cc:385] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:36.169898 10013 raft_consensus.cc:740] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:36.170424 10013 consensus_queue.cc:260] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.179476 9932 consensus_queue.cc:260] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.189697 10013 ts_tablet_manager.cc:1434] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.024s user 0.001s sys 0.005s
I20260504 13:56:36.190737 10013 tablet_bootstrap.cc:492] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:36.194425 9932 ts_tablet_manager.cc:1434] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.061s user 0.006s sys 0.000s
I20260504 13:56:36.197567 9932 tablet_bootstrap.cc:492] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:36.219645 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_2f0fc59ee89c402cb8a0fc23341636e6 [id=00b5922717934e15993bafde9f2513e7]
I20260504 13:56:36.222205 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_798d2540b8f4489fb11763ccaa36bced [id=049b4fa786fc4dbb8e52931dcaaab90f]
I20260504 13:56:36.222420 10810 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Refusing update from remote peer 806407748d564439807fe5d73db9abda: Log matching property violated. Preceding OpId in replica: term: 3 index: 351. Preceding OpId from leader: term: 4 index: 355. (index mismatch)
I20260504 13:56:36.224555 11140 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [LEADER]: Connected to new peer: Peer: permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 349, Time since last communication: 0.000s
I20260504 13:56:36.242933 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_512ae07c2df74cb99bb77006d985f2da [id=09959f81c69c49778a6ee33dc1e85aae]
I20260504 13:56:36.243861 9932 tablet_bootstrap.cc:654] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:36.249413 9931 tablet_bootstrap.cc:492] T 2f6b3e969760442283c3191e062c1839 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:36.249882 10956 raft_consensus.cc:493] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:36.250305 10956 raft_consensus.cc:515] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.249958 9931 ts_tablet_manager.cc:1403] T 2f6b3e969760442283c3191e062c1839 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.224s user 0.011s sys 0.000s
I20260504 13:56:36.252259 10956 leader_election.cc:290] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:36.261427 10013 tablet_bootstrap.cc:654] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:36.254068 9931 raft_consensus.cc:359] T 2f6b3e969760442283c3191e062c1839 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.270430 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba4584755aae47e0b0f14c6c55474d18" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:36.269456 9931 raft_consensus.cc:385] T 2f6b3e969760442283c3191e062c1839 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:36.271644 9931 raft_consensus.cc:740] T 2f6b3e969760442283c3191e062c1839 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
W20260504 13:56:36.271957 9474 leader_election.cc:343] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:36.286017 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba4584755aae47e0b0f14c6c55474d18" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:36.290634 9683 raft_consensus.cc:2468] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:36.292037 9473 leader_election.cc:304] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:36.309007 9932 tablet_bootstrap.cc:492] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:36.309409 9932 ts_tablet_manager.cc:1403] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.112s user 0.011s sys 0.000s
I20260504 13:56:36.272540 9931 consensus_queue.cc:260] T 2f6b3e969760442283c3191e062c1839 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.316059 11060 raft_consensus.cc:2804] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:36.316337 9931 ts_tablet_manager.cc:1434] T 2f6b3e969760442283c3191e062c1839 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.065s user 0.007s sys 0.000s
I20260504 13:56:36.316406 11060 raft_consensus.cc:493] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:36.316706 11060 raft_consensus.cc:3060] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:36.331017 11172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 806407748d564439807fe5d73db9abda. Latest consensus state: current_term: 4 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:36.333873 11172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:56:36.339111 11060 raft_consensus.cc:515] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.341111 11060 leader_election.cc:290] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:36.342166 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba4584755aae47e0b0f14c6c55474d18" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:36.342677 9681 raft_consensus.cc:3060] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:36.343004 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba4584755aae47e0b0f14c6c55474d18" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:36.344254 11172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:36.344906 11172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: This master's current role is: FOLLOWER
W20260504 13:56:36.347397 9474 leader_election.cc:343] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:36.348039 10013 tablet_bootstrap.cc:492] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:36.348909 10013 ts_tablet_manager.cc:1403] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.158s user 0.008s sys 0.003s
I20260504 13:56:36.351330 10013 raft_consensus.cc:359] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.359524 9681 raft_consensus.cc:2468] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:36.342290 11180 tablet.cc:1724] T 274bb204041c4e1085395c7bd938aaf1 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260504 13:56:36.361104 9473 leader_election.cc:304] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:36.363101 11135 raft_consensus.cc:2804] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:36.363263 10013 raft_consensus.cc:385] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:36.363600 10013 raft_consensus.cc:740] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:36.363700 11135 raft_consensus.cc:697] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:36.364588 11135 consensus_queue.cc:237] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.340539 11177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:36.366426 11177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:56:36.367379 9832 raft_consensus.cc:3060] T 2f6b3e969760442283c3191e062c1839 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:36.372839 9932 raft_consensus.cc:359] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.373474 9932 raft_consensus.cc:385] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:36.373760 9932 raft_consensus.cc:740] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:36.374393 9932 consensus_queue.cc:260] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.375366 10013 consensus_queue.cc:260] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.377316 10013 ts_tablet_manager.cc:1434] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.028s user 0.004s sys 0.000s
I20260504 13:56:36.380281 9931 tablet_bootstrap.cc:492] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:36.392750 11146 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:36.393136 11146 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:36.394815 11146 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:56:36.399935 10013 tablet_bootstrap.cc:492] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:36.403519 11139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:36.404207 11139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: This master's current role is: LEADER
I20260504 13:56:36.408669 10810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:36.424609 10013 tablet_bootstrap.cc:654] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:36.426594 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38]
I20260504 13:56:36.451323 9932 ts_tablet_manager.cc:1434] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.142s user 0.009s sys 0.000s
I20260504 13:56:36.454979 10956 raft_consensus.cc:493] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:36.455557 10956 raft_consensus.cc:515] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.455945 11189 tablet.cc:1724] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:36.457592 10956 leader_election.cc:290] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:36.458508 11187 raft_consensus.cc:493] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:36.458957 11187 raft_consensus.cc:515] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.459486 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a1a77ad56634e068401a2bfbc977761" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:36.459896 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:36.460763 11187 leader_election.cc:290] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:36.467586 11183 tablet.cc:1724] T 274bb204041c4e1085395c7bd938aaf1 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260504 13:56:36.474552 9679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a1a77ad56634e068401a2bfbc977761" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:36.475343 9679 raft_consensus.cc:2393] T 4a1a77ad56634e068401a2bfbc977761 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 1: Already voted for candidate d4051d2bf0bf410eb8acd107560df888 in this term.
I20260504 13:56:36.477288 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ffeae4e5af4cc8a3eb3bc0bedf18cc" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:36.479560 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ffeae4e5af4cc8a3eb3bc0bedf18cc" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:36.480890 9618 leader_election.cc:343] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:36.483482 11180 tablet.cc:1724] T 7eaa40eb4b1c425a9edfb7eaba953b65 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:36.491240 11135 consensus_queue.cc:1048] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:36.497627 9473 leader_election.cc:304] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:36.498982 10956 raft_consensus.cc:3060] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:36.506649 9932 tablet_bootstrap.cc:492] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:36.511350 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_dd11ea89ad7248c39ec114af58cf598f [id=14bd1983ba95468e9e70d8f5724f42e6]
I20260504 13:56:36.517071 11190 tablet.cc:1724] T 7eaa40eb4b1c425a9edfb7eaba953b65 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:36.520499 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_2bed5ea8f43149e8bc3b21cd9d1b3846 [id=15994ccc978545058e324152f70c8c3e]
W20260504 13:56:36.523691 9330 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 consensus state has no leader
W20260504 13:56:36.527756 9620 leader_election.cc:343] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:36.528170 9620 leader_election.cc:304] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:36.477542 9931 tablet_bootstrap.cc:654] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:36.534132 11191 tablet.cc:1724] T 274bb204041c4e1085395c7bd938aaf1 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260504 13:56:36.506595 10956 raft_consensus.cc:2749] T 4a1a77ad56634e068401a2bfbc977761 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:36.548363 9932 tablet_bootstrap.cc:654] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:36.548750 11181 raft_consensus.cc:2749] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:36.550092 11187 raft_consensus.cc:493] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:36.550498 11187 raft_consensus.cc:515] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:36.552268 11187 leader_election.cc:290] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:36.555452 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af5c5a6cae4e4b6ea3eece2e21a587ec" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:36.558298 11181 raft_consensus.cc:493] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:36.562794 11135 consensus_queue.cc:1048] T 2f6b3e969760442283c3191e062c1839 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:36.562971 9618 leader_election.cc:343] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:36.564929 10013 tablet_bootstrap.cc:492] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
W20260504 13:56:36.564867 10690 leader_election.cc:336] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781): Timed out: RequestConsensusVote RPC to 127.8.238.252:44781 timed out after 0.573s (SENT)
I20260504 13:56:36.565308 10013 ts_tablet_manager.cc:1403] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.166s user 0.009s sys 0.005s
I20260504 13:56:36.565305 10690 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:36.566365 11177 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:56:36.568078 10013 raft_consensus.cc:359] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.568708 10013 raft_consensus.cc:385] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:36.568980 10013 raft_consensus.cc:740] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:36.558753 11181 raft_consensus.cc:515] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.569582 10013 consensus_queue.cc:260] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.571878 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cb30835150949f39f6e7e7563381763" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:36.573194 9618 leader_election.cc:343] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:36.573944 11135 raft_consensus.cc:493] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:36.574410 11135 raft_consensus.cc:515] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.577773 11135 leader_election.cc:290] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:36.579090 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_dee1335eff89438e9c68835a4c258031 [id=1714df2b31ff4865b6b357500d121a26]
I20260504 13:56:36.581403 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_24398dd8ee63413ebb585587353ea155 [id=1d2bbdaf6382433cb8c2a55f62f0388d]
I20260504 13:56:36.594609 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a470ddc5bcb4121bd2f5f02f6e6b945" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:36.595135 9683 raft_consensus.cc:2468] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:36.595981 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af5c5a6cae4e4b6ea3eece2e21a587ec" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:36.597299 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a470ddc5bcb4121bd2f5f02f6e6b945" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:36.604154 9474 leader_election.cc:343] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:36.605726 10013 ts_tablet_manager.cc:1434] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20260504 13:56:36.607367 11181 leader_election.cc:290] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:36.611299 9473 leader_election.cc:304] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:36.612227 10956 raft_consensus.cc:2804] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:36.614195 10956 raft_consensus.cc:493] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:36.604954 11158 consensus_queue.cc:1048] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:36.605155 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cb30835150949f39f6e7e7563381763" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:36.617777 11158 raft_consensus.cc:493] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 13:56:36.618584 9620 leader_election.cc:343] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:36.618736 11158 raft_consensus.cc:515] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.621380 11158 leader_election.cc:290] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:36.621286 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de6714ee86d64112baf13dd1b2b87582" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:36.621920 9533 raft_consensus.cc:2468] T de6714ee86d64112baf13dd1b2b87582 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:36.622375 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de6714ee86d64112baf13dd1b2b87582" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:36.622951 9681 raft_consensus.cc:2468] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:36.623010 9767 leader_election.cc:304] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:36.624047 11158 raft_consensus.cc:2804] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:36.625173 11158 raft_consensus.cc:493] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:36.625533 11158 raft_consensus.cc:3060] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:36.626487 10956 raft_consensus.cc:3060] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:36.628939 9932 tablet_bootstrap.cc:492] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:36.629334 9932 ts_tablet_manager.cc:1403] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.123s user 0.009s sys 0.000s
I20260504 13:56:36.632920 9932 raft_consensus.cc:359] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:36.612807 9176 leader_election.cc:336] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781): Timed out: RequestConsensusVote RPC to 127.8.238.252:44781 timed out after 0.533s (SENT)
I20260504 13:56:36.634888 9932 raft_consensus.cc:385] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:36.635111 9932 raft_consensus.cc:740] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:36.618953 9620 leader_election.cc:304] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:36.639240 11187 raft_consensus.cc:2749] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:36.640957 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_60fe6ee7a2414e2aa8ea2d57ee4a827d [id=1de44a1bbf06492084cfd70cf540b213]
W20260504 13:56:36.641600 9620 leader_election.cc:343] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:36.643778 10013 tablet_bootstrap.cc:492] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:36.635689 9932 consensus_queue.cc:260] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:36.647408 9620 leader_election.cc:304] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:36.666862 11187 raft_consensus.cc:2749] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:36.672780 10956 raft_consensus.cc:515] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.653851 9932 ts_tablet_manager.cc:1434] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260504 13:56:36.687688 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]
I20260504 13:56:36.689227 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_5ff6dab0e47d41e681533fca02a36153 [id=230a8885dd2b4995815b650d3d9f9f75]
I20260504 13:56:36.718339 9932 tablet_bootstrap.cc:492] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:36.725093 11158 raft_consensus.cc:515] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.727876 11187 raft_consensus.cc:493] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:36.728585 11158 leader_election.cc:290] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:36.728297 11187 raft_consensus.cc:515] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.730018 11187 leader_election.cc:290] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:36.736153 9932 tablet_bootstrap.cc:654] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:36.742347 11187 raft_consensus.cc:493] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:36.742754 11187 raft_consensus.cc:515] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:36.746569 11187 leader_election.cc:290] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:36.772393 9931 tablet_bootstrap.cc:492] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:36.772548 9932 tablet_bootstrap.cc:492] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:36.772450 11010 consensus_queue.cc:1048] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:36.772851 9931 ts_tablet_manager.cc:1403] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.456s user 0.014s sys 0.002s
I20260504 13:56:36.772930 9932 ts_tablet_manager.cc:1403] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.000s
I20260504 13:56:36.776309 9931 raft_consensus.cc:359] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.776906 9931 raft_consensus.cc:385] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:36.783787 9932 raft_consensus.cc:359] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.784425 9932 raft_consensus.cc:385] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:36.784689 9932 raft_consensus.cc:740] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:36.785262 9932 consensus_queue.cc:260] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.692523 10013 tablet_bootstrap.cc:654] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:36.727721 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de6714ee86d64112baf13dd1b2b87582" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:36.733251 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a89bac060214d30924398007dfa1fa1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:36.790872 9533 raft_consensus.cc:3060] T de6714ee86d64112baf13dd1b2b87582 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:36.745834 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ddf890c17964e7f90409ebe3c4034ef" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:36.750622 11026 consensus_queue.cc:1048] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:36.791962 9618 leader_election.cc:343] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:36.792528 9618 leader_election.cc:343] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:36.794624 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 133150us
I20260504 13:56:36.795840 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de6714ee86d64112baf13dd1b2b87582" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:36.796411 9681 raft_consensus.cc:3060] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:36.799127 9533 raft_consensus.cc:2468] T de6714ee86d64112baf13dd1b2b87582 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:36.800376 9767 leader_election.cc:304] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:36.802670 11010 raft_consensus.cc:2804] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:36.803077 11010 raft_consensus.cc:697] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:36.803862 11010 consensus_queue.cc:237] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.816921 9932 ts_tablet_manager.cc:1434] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20260504 13:56:36.821398 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a470ddc5bcb4121bd2f5f02f6e6b945" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:36.822448 9474 leader_election.cc:343] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:36.829155 10013 tablet_bootstrap.cc:492] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:36.829576 10013 ts_tablet_manager.cc:1403] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.186s user 0.009s sys 0.001s
I20260504 13:56:36.832181 10956 leader_election.cc:290] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:36.833343 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a470ddc5bcb4121bd2f5f02f6e6b945" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:36.833969 9683 raft_consensus.cc:3060] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:36.837384 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_09255abca80f41dc895d81ccd89b9c7a [id=24485cf365ad4f9b89fa0f716fdb34cd]
I20260504 13:56:36.839196 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee]
I20260504 13:56:36.839965 11199 consensus_queue.cc:1048] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:36.841192 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_2585508fc4214c8b82d2957644309ac2 [id=256b605e5a2342ed883ec30e3de0ffda]
I20260504 13:56:36.850569 11158 consensus_queue.cc:1048] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:36.854089 9683 raft_consensus.cc:2468] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:36.856230 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a89bac060214d30924398007dfa1fa1" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:36.857631 9473 leader_election.cc:304] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:36.858954 11026 raft_consensus.cc:2804] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:36.859421 11026 raft_consensus.cc:697] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:36.860666 11026 consensus_queue.cc:237] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:36.861761 9474 consensus_peers.cc:597] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:36.863901 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ddf890c17964e7f90409ebe3c4034ef" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:36.865456 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 58076.8us
I20260504 13:56:36.777144 9931 raft_consensus.cc:740] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:36.872478 9931 consensus_queue.cc:260] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.883667 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_42f086bc327c42ccad96925c681a386a [id=2c6aa17153154189a33f5f3aa00b62e9]
I20260504 13:56:36.888314 9932 tablet_bootstrap.cc:492] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:36.893694 9932 tablet_bootstrap.cc:654] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:36.899544 11187 raft_consensus.cc:493] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 13:56:36.900162 9620 leader_election.cc:343] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:36.900028 11187 raft_consensus.cc:515] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.900548 9620 leader_election.cc:304] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:36.901422 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_69c9551ae2cf4ef399e40017a33bad73 [id=2d9b8dca85e9477eb77877d5618e9b6f]
I20260504 13:56:36.901798 11187 leader_election.cc:290] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:36.902680 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc2f95dadd544f57bd6d1467d30e954d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:36.904044 9618 leader_election.cc:343] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:36.915876 11181 raft_consensus.cc:2749] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:36.928385 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e]
I20260504 13:56:36.931442 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_1611aa9f4dba40e09dae4433e2d20154 [id=2fb2c56e2fb44b9aac476e06e36e5778]
I20260504 13:56:36.939785 11158 raft_consensus.cc:493] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:36.940191 11158 raft_consensus.cc:515] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.941834 11158 leader_election.cc:290] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:36.942508 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_2d5a582adb634895a55193dd861f3b60 [id=30f349197d564e8da82855fec17b4cd2]
I20260504 13:56:36.943257 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e86027e3e74bc196e9a9ffc9fdf8a5" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:36.947186 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]
I20260504 13:56:36.953737 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f]
I20260504 13:56:36.961205 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_963c93d5d8ed49e6a2052fa1e652da91 [id=399a65bb2fa6428296341de74e964518]
I20260504 13:56:36.962675 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_f8a23a8252f84cacb13b338f9d9dc9fc [id=39be2c4d16aa41c0832bfeb23269243a]
I20260504 13:56:36.966536 9931 ts_tablet_manager.cc:1434] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.193s user 0.006s sys 0.001s
I20260504 13:56:36.967466 9931 tablet_bootstrap.cc:492] T 2a629c2a69084b6b98984c8d374eeb1e P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:36.831877 10013 raft_consensus.cc:359] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.966595 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_073a6dae9da444bea1f96b2f33fe1a85 [id=39c692c399dd43ca963357b6818871cf]
I20260504 13:56:36.967741 10013 raft_consensus.cc:385] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:36.967998 10013 raft_consensus.cc:740] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:36.968621 10013 consensus_queue.cc:260] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:36.971019 10013 ts_tablet_manager.cc:1434] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.141s user 0.005s sys 0.000s
W20260504 13:56:36.945931 9620 leader_election.cc:343] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:36.971863 9620 leader_election.cc:304] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:36.973490 11187 raft_consensus.cc:2749] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:36.979079 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc2f95dadd544f57bd6d1467d30e954d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:36.986583 11135 consensus_queue.cc:1048] T a26c6a2ffb8746a3941a56c7259412e3 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:36.990377 9680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3e86027e3e74bc196e9a9ffc9fdf8a5" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:36.991086 9680 raft_consensus.cc:2393] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:36.992230 9768 leader_election.cc:304] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:36.993323 11206 raft_consensus.cc:3060] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:36.970499 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_4207c9e8019c4088b934376c5568d101 [id=3ca89308de104a5fa27cac4c62047304]
I20260504 13:56:37.003845 11206 raft_consensus.cc:2749] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:37.006709 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]
I20260504 13:56:37.008210 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_5944297f8c104978bbc6913ecd8f47a4 [id=3f11131cece44f3ebf3cd0f1e69139ae]
I20260504 13:56:37.009605 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]
I20260504 13:56:37.009745 11060 raft_consensus.cc:493] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:37.010193 11060 raft_consensus.cc:515] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:37.012068 9331 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.238.254:36145 timed out after 0.616s (SENT) [suppressed 6 similar messages]
I20260504 13:56:37.015844 11060 leader_election.cc:290] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:37.014181 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dffddd4b89984c54bc26dcb4a9488525" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:37.022989 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:37.011030 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_1665799e268a496d9ffc5568fa34a763 [id=403cbbc8a1304e5695a89d3697932c26]
W20260504 13:56:37.025426 9474 leader_election.cc:343] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:37.027539 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_003cbc6fdf0b4dddaba008af167dc621 [id=4629003a2d8d461c9e309d49ddecea6a]
I20260504 13:56:37.032596 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_a660204a2583425aa3ecd09c5cc01368 [id=4683a87561f14c049c1ed8eb68dc24d1]
I20260504 13:56:36.966897 9681 raft_consensus.cc:2468] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:37.046196 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_821e9f466c974a2fbf24c07f8c266e62 [id=469bd5cb4e7f4a8a9c7af947a834ca7b]
I20260504 13:56:37.060463 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_f7b06aadc0024855b08b56cad8c7b6a3 [id=4efd9dd69847446d9f05700d3bd3f914]
W20260504 13:56:37.069734 9331 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.254:36145, user_credentials={real_user=slave}} blocked reactor thread for 57683.3us
I20260504 13:56:37.071031 9931 tablet_bootstrap.cc:654] T 2a629c2a69084b6b98984c8d374eeb1e P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:37.067763 9176 leader_election.cc:336] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781): Timed out: RequestConsensusVote RPC to 127.8.238.252:44781 timed out after 0.678s (SENT)
I20260504 13:56:37.095489 10956 consensus_queue.cc:1048] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:37.098245 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]
I20260504 13:56:37.114392 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_2d928cb3553f40bc8e825d0741ef3199 [id=51b254be98ee457d892039784e4a5f0b]
I20260504 13:56:37.123775 9932 tablet_bootstrap.cc:492] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:37.125346 9932 ts_tablet_manager.cc:1403] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.237s user 0.023s sys 0.043s
I20260504 13:56:37.126125 10013 tablet_bootstrap.cc:492] T 89edd71cf5aa4a6abe2084695d3ed807 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:37.142874 9679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dffddd4b89984c54bc26dcb4a9488525" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:37.143570 9679 raft_consensus.cc:2468] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
W20260504 13:56:37.145159 9620 leader_election.cc:343] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:37.136175 9932 raft_consensus.cc:359] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.164005 9620 leader_election.cc:304] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:37.164021 9932 raft_consensus.cc:385] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:37.164990 11187 raft_consensus.cc:2749] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:37.169773 9473 leader_election.cc:304] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:37.170776 11060 raft_consensus.cc:2804] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:37.171355 11060 raft_consensus.cc:493] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:37.171784 11060 raft_consensus.cc:3060] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:37.167905 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]
I20260504 13:56:37.165544 9932 raft_consensus.cc:740] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:37.181869 9932 consensus_queue.cc:260] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.183830 9932 ts_tablet_manager.cc:1434] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.058s user 0.007s sys 0.000s
I20260504 13:56:37.184656 9932 tablet_bootstrap.cc:492] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:37.197151 11060 raft_consensus.cc:515] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.199721 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_af058f50341e4f478c970f8cb52bf0d7 [id=57ccd7218d454c8c908fe06b798bd83b]
I20260504 13:56:37.217890 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_597b4ca3e37849d0aa9c86eed64d0614 [id=5a597e1cae1d4126ae355289aef1ced6]
I20260504 13:56:37.221957 9932 tablet_bootstrap.cc:654] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:37.226831 9931 tablet_bootstrap.cc:492] T 2a629c2a69084b6b98984c8d374eeb1e P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:37.227253 9931 ts_tablet_manager.cc:1403] T 2a629c2a69084b6b98984c8d374eeb1e P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.260s user 0.010s sys 0.000s
I20260504 13:56:37.229696 9931 raft_consensus.cc:359] T 2a629c2a69084b6b98984c8d374eeb1e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.230288 9931 raft_consensus.cc:385] T 2a629c2a69084b6b98984c8d374eeb1e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:37.230533 9931 raft_consensus.cc:740] T 2a629c2a69084b6b98984c8d374eeb1e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:37.242069 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62]
I20260504 13:56:37.250146 11187 raft_consensus.cc:493] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:37.250622 11187 raft_consensus.cc:515] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.252007 9932 tablet_bootstrap.cc:492] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:37.252441 9932 ts_tablet_manager.cc:1403] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.002s
I20260504 13:56:37.252949 11187 leader_election.cc:290] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:37.255686 9932 raft_consensus.cc:359] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.255911 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e53381ee0e7f4e9781b833b435bca831" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:37.256309 9932 raft_consensus.cc:385] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:37.256567 9932 raft_consensus.cc:740] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:37.257157 9932 consensus_queue.cc:260] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.257869 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_18f9a8030cea4cc0abedb81cd5accc59 [id=5c4f226dc33b428e92ca77959e075d7c]
W20260504 13:56:37.259714 9620 leader_election.cc:343] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:37.264217 9932 ts_tablet_manager.cc:1434] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260504 13:56:37.267624 9932 tablet_bootstrap.cc:492] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:37.269403 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_2099388b6e2a455295be76a6d072758e [id=63e34a66cd644ab7bede27373e321714]
I20260504 13:56:37.284999 9932 tablet_bootstrap.cc:654] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:37.231112 9931 consensus_queue.cc:260] T 2a629c2a69084b6b98984c8d374eeb1e P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.289551 9931 ts_tablet_manager.cc:1434] T 2a629c2a69084b6b98984c8d374eeb1e P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.062s user 0.005s sys 0.000s
I20260504 13:56:37.201773 10013 tablet_bootstrap.cc:654] T 89edd71cf5aa4a6abe2084695d3ed807 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:37.223256 10956 raft_consensus.cc:493] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:37.300092 10956 raft_consensus.cc:515] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.301983 10956 leader_election.cc:290] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:37.299367 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_9bc232a8e4dd4df1a36b35d09dd50d0a [id=66faab34fb324213a874ea6fae3c96a1]
I20260504 13:56:37.254328 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e53381ee0e7f4e9781b833b435bca831" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:37.303661 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a629c2a69084b6b98984c8d374eeb1e" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:37.304085 9618 leader_election.cc:343] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:37.304234 9828 raft_consensus.cc:2468] T 2a629c2a69084b6b98984c8d374eeb1e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:37.304452 9618 leader_election.cc:304] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:37.305397 11187 raft_consensus.cc:2749] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:37.305423 9474 leader_election.cc:304] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:37.306318 10956 raft_consensus.cc:2804] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:37.306720 10956 raft_consensus.cc:493] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:37.307137 10956 raft_consensus.cc:3060] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:37.308887 9679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a629c2a69084b6b98984c8d374eeb1e" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:37.309481 9679 raft_consensus.cc:2468] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:37.309252 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_0134ffe7086440ccbcadd076d8fe5d01 [id=693e9b6ed89b4ab7b9d5dc7402e17f20]
I20260504 13:56:37.312417 11060 leader_election.cc:290] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:37.312762 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dffddd4b89984c54bc26dcb4a9488525" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:37.313473 9677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dffddd4b89984c54bc26dcb4a9488525" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:37.313997 9677 raft_consensus.cc:3060] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:37.314064 9474 leader_election.cc:343] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:37.317076 10956 raft_consensus.cc:515] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.319586 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a629c2a69084b6b98984c8d374eeb1e" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:37.320163 9828 raft_consensus.cc:3060] T 2a629c2a69084b6b98984c8d374eeb1e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:37.320587 9677 raft_consensus.cc:2468] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:37.321403 9932 tablet_bootstrap.cc:492] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:37.321818 9932 ts_tablet_manager.cc:1403] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.007s
I20260504 13:56:37.323527 9473 leader_election.cc:304] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:37.324206 9932 raft_consensus.cc:359] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.324833 9932 raft_consensus.cc:385] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:37.325099 9932 raft_consensus.cc:740] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:37.325754 11135 raft_consensus.cc:2804] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:37.325654 9932 consensus_queue.cc:260] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.326535 11135 raft_consensus.cc:697] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:37.328799 9828 raft_consensus.cc:2468] T 2a629c2a69084b6b98984c8d374eeb1e P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:37.328929 9932 ts_tablet_manager.cc:1434] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260504 13:56:37.327791 11135 consensus_queue.cc:237] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.329761 9474 leader_election.cc:304] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:37.330416 9932 tablet_bootstrap.cc:492] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:37.330410 11026 raft_consensus.cc:2804] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:37.330832 11026 raft_consensus.cc:697] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:37.331652 11026 consensus_queue.cc:237] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.336366 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568]
I20260504 13:56:37.338562 11181 raft_consensus.cc:493] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:37.339020 11181 raft_consensus.cc:515] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.340929 11181 leader_election.cc:290] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:37.342226 9932 tablet_bootstrap.cc:654] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:37.345904 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_f00a5967b1b7422b830dcf11db2acbd8 [id=6a34d9af17c84984b479bbd10025fba5]
I20260504 13:56:37.370919 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_5636294717ff4c63939c2cf9facfc788 [id=6acf451855024161b7826094f8387a9c]
I20260504 13:56:37.384752 10956 leader_election.cc:290] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:37.384940 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_93e03451ea09440d847c57c026230518 [id=6ad5facbe9ca40eeb7746baef5c566a7]
I20260504 13:56:37.396813 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36c8e9ba29a1466082a3e8bedec87b91" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:37.397917 11181 raft_consensus.cc:493] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 13:56:37.398836 9620 leader_election.cc:343] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:37.398406 11181 raft_consensus.cc:515] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.401827 11181 leader_election.cc:290] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:37.400904 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_8ccb9fc27b374611b20008514d3ad483 [id=6c295b6b72c94ca0b5f1805a555e7f77]
I20260504 13:56:37.409572 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_780e641042db4fd5aeb8af8b1aa08c72 [id=6cb26719d5fe45ccbdef18571c902228]
I20260504 13:56:37.411149 11181 raft_consensus.cc:493] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:37.411657 11181 raft_consensus.cc:515] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.414162 11181 leader_election.cc:290] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:37.415185 11187 raft_consensus.cc:493] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:37.415596 11187 raft_consensus.cc:515] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.417548 11187 leader_election.cc:290] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:37.418505 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "538c8c2412104c3e8b19394a1cc38818" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:37.418826 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "982f7c53a1f0482e862b53aeeee7cd25" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:37.419871 9620 leader_election.cc:343] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:37.420509 9620 leader_election.cc:343] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:37.421386 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_6462cb42c34a4eb8976d8d2a4f82d434 [id=6e3151859b6d4da3a477367687862671]
I20260504 13:56:37.430562 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71f444ce9214e3f98eff6ac8ca7dde7" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:37.435490 9620 leader_election.cc:343] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:37.434857 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_49715b69b840473b9f7b0fdabcc0084e [id=700b914cd7ec44e791cd0a27d4b9bf81]
I20260504 13:56:37.445813 9931 tablet_bootstrap.cc:492] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:37.446108 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_11d8ab61a6044154bb599875a0bc9004 [id=73e12524eb9e416782bf9f7fb58c93bd]
I20260504 13:56:37.452761 9931 tablet_bootstrap.cc:654] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:37.465267 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_7dbd11258f34498e82a86f7e3c72f470 [id=7bf52f9bd2a042e59fc4ff612d9846e9]
I20260504 13:56:37.476737 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_df8874a20e834872a12300a4f5628273 [id=7fc0413d79184a2dbe5c470b8506b4e1]
I20260504 13:56:37.487681 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_4842f50414d34bdb8a10802bc059b421 [id=7fc23620ef7342eea52bcce264eb12b3]
I20260504 13:56:37.504381 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_24dab2dd1f45402786d80cba80bbad82 [id=80ca58df22214450b894afea377e9c16]
I20260504 13:56:37.509003 9679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a629c2a69084b6b98984c8d374eeb1e" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:37.509490 9932 tablet_bootstrap.cc:492] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:37.509615 9679 raft_consensus.cc:3060] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:37.509934 9932 ts_tablet_manager.cc:1403] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.180s user 0.039s sys 0.030s
I20260504 13:56:37.512367 9932 raft_consensus.cc:359] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.512988 9932 raft_consensus.cc:385] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:37.513269 9932 raft_consensus.cc:740] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:37.514879 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_cdee25533fe54da0a74b421714b0d7c4 [id=81df524d85774544807ba1902be1aae1]
W20260504 13:56:37.518635 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:37.519017 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36c8e9ba29a1466082a3e8bedec87b91" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:37.519893 9932 consensus_queue.cc:260] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.521790 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71f444ce9214e3f98eff6ac8ca7dde7" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:37.523201 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "538c8c2412104c3e8b19394a1cc38818" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:37.525427 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]
I20260504 13:56:37.517803 9679 raft_consensus.cc:2468] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:37.534173 10013 tablet_bootstrap.cc:492] T 89edd71cf5aa4a6abe2084695d3ed807 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:37.534416 9932 ts_tablet_manager.cc:1434] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260504 13:56:37.534844 10013 ts_tablet_manager.cc:1403] T 89edd71cf5aa4a6abe2084695d3ed807 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.409s user 0.014s sys 0.000s
W20260504 13:56:37.548915 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 53900us
I20260504 13:56:37.557016 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_1c9b2e63142440a5a45018d187a6cded [id=8a427e40b3374dafa140c9156edcc660]
I20260504 13:56:37.558763 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_9e9ef3353dcf4b1fadaa2f51bc8f7041 [id=8dcabd7d67334598926f927a62e630d6]
I20260504 13:56:37.560308 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_b59e5e285d50401e94d51afeb4cbc500 [id=8f16af9aec63405cbae2a91b84fa3009]
I20260504 13:56:37.562624 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_a992fb6e038e4408a781d6d8ead37f54 [id=937a02c37dbd45d5bc92979ceaee1c06]
I20260504 13:56:37.564740 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]
I20260504 13:56:37.565996 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_443402c40ae5443daf4ba4258b847303 [id=98126e5c27cd4d3da57fcd88d481de79]
I20260504 13:56:37.568233 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_10e92469b9cb4e9b965d0a10cfeecca8 [id=99da5017cead4ddeaa3441064b1ed716]
I20260504 13:56:37.569536 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_655e2e8cf3544fbeb0ce33829bb2c8a0 [id=9db1e031520d405ea4989e30d5194911]
I20260504 13:56:37.570899 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4]
I20260504 13:56:37.572942 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_225ad80b8318432b88a729064c88e8ec [id=a3cca003025745cebf8e7d403d0778cf]
I20260504 13:56:37.573475 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "982f7c53a1f0482e862b53aeeee7cd25" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:37.574944 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_e55147137c794be4bff21afce5d36bbc [id=b45733fe3c5547bea00e3967be81ca35]
I20260504 13:56:37.576887 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_6e41cb79b95848ba8c7b1f05f961c7f3 [id=b91fe7c5c3844c109756842f93098699]
I20260504 13:56:37.578228 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]
I20260504 13:56:37.579531 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_862ae682dfb44737a72e7c5df628a957 [id=bbef146162cb4c30a3ba42ca2a36fe9f]
I20260504 13:56:37.581267 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_5bc32a610edd49ab9641c3726f53f5c6 [id=be5c761518674e7782bfff73366b1675]
W20260504 13:56:37.582149 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 51647.6us
I20260504 13:56:37.583765 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_e116f9abbb23419d9a3365f7d0e17db9 [id=bee8acaa102249578514a3d4e5f876d3]
I20260504 13:56:37.585062 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]
I20260504 13:56:37.586087 11187 raft_consensus.cc:493] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:37.586421 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_849bbe7902ca44969c5208828774fa06 [id=c146e84e0e844deda9af40562f980047]
I20260504 13:56:37.586575 11187 raft_consensus.cc:515] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.589212 11187 leader_election.cc:290] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:37.590456 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_861ff05bb9e144ea85a3fb7d75f68749 [id=c4b5a9431b8b4d33b41fa27a357e1f46]
I20260504 13:56:37.591785 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_363d6634e42e4dd6a5686c1f0e401ef2 [id=d0d8b54fb69540589420b87f90b4c823]
I20260504 13:56:37.593132 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_3387191fcd6046abbf257447bfbb27f7 [id=d195f5bef6ce49f1b87a4bf5f516eebb]
I20260504 13:56:37.595158 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469]
W20260504 13:56:37.596585 9618 leader_election.cc:343] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:37.596969 9618 leader_election.cc:304] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:37.598076 11187 raft_consensus.cc:2749] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:37.603444 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_6b0124eccb02454b9195b8ddaae5bd38 [id=d47a2c6a9cb34ba4ade10fc23913bf7d]
I20260504 13:56:37.604764 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_d6a5a22d78264e799e80d31d5749600a [id=d63acf266ca440729128a717eed32433]
I20260504 13:56:37.609149 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_91cc6c38c9ff43dca25814664068b1c6 [id=da5c427e75e74ea1ae3a5961ed27e0de]
I20260504 13:56:37.613026 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_8cae213b77994b9fba63b8a7c5f27249 [id=df86bccb4ca8414c90f9fd8753724d89]
I20260504 13:56:37.614323 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1]
I20260504 13:56:37.615644 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_06ecf48a04d94ac8a7e954ff2c260dbf [id=e11be55403d24c8bb35f4708da2e9a78]
I20260504 13:56:37.616932 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_999e37d0dab54ca68c0fa66f74582109 [id=e37fd7e36e604a60832acf303f0ecb7f]
I20260504 13:56:37.618201 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_89c38f9ba46e48fab397a7f3d5d1b805 [id=e4f6621892874f979d1c579d487cba44]
I20260504 13:56:37.620235 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]
W20260504 13:56:37.621327 9618 leader_election.cc:343] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:37.621510 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_737cc12f07144a6c92dc2974693a649d [id=e9103e8b725b4e81b6cf094395beef3b]
I20260504 13:56:37.621690 9618 leader_election.cc:304] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:37.622627 11187 raft_consensus.cc:2749] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:37.623534 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_a5fa0a0bead94aa3b02d6175fefde171 [id=e9fbd43d30ae4d1dbeea534025125b91]
I20260504 13:56:37.624815 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_1e6b4c9e684e45b3b2510d3979c1b5c4 [id=ecfa480ab0ea40008ff4507ed6c5b964]
I20260504 13:56:37.626381 9932 tablet_bootstrap.cc:492] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:37.626818 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_2dc0381bd6c84ab59d5bb422de89c0e2 [id=ed756f69850642188b881bd3c680a622]
I20260504 13:56:37.628077 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_09fd612a3c4a4303b8f05530e7d49819 [id=eda14d7d5f7d4391a8c470677ffab4ea]
I20260504 13:56:37.629339 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_72e3a254561b4cae927e6b6bcacd87b6 [id=f227576f105c4b1fbd459b3638ce684b]
I20260504 13:56:37.629391 9931 tablet_bootstrap.cc:492] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:37.629925 9931 ts_tablet_manager.cc:1403] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.184s user 0.009s sys 0.002s
I20260504 13:56:37.630690 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_7f912e1d4381428e9572cbb4b13f4eec [id=fb18c55d21704b2ab04dbf35d21377bf]
I20260504 13:56:37.631955 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_9b40ba316993413181a9743fb7bab7fb [id=fb4e5ce5526e44de8effdd766f4c257f]
I20260504 13:56:37.632776 11187 raft_consensus.cc:493] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:37.634199 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_7db59fa280b6442b99cf24677b57df83 [id=fdb915ebe83443fd8d2f7f287385f6a4]
I20260504 13:56:37.633311 11187 raft_consensus.cc:515] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.635455 11155 catalog_manager.cc:679] Loaded metadata for table default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]
I20260504 13:56:37.636626 11187 leader_election.cc:290] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:37.644282 9932 tablet_bootstrap.cc:654] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:37.651130 10013 raft_consensus.cc:359] T 89edd71cf5aa4a6abe2084695d3ed807 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.651993 10013 raft_consensus.cc:385] T 89edd71cf5aa4a6abe2084695d3ed807 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:37.652323 10013 raft_consensus.cc:740] T 89edd71cf5aa4a6abe2084695d3ed807 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:37.653218 10013 consensus_queue.cc:260] T 89edd71cf5aa4a6abe2084695d3ed807 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:37.654469 9618 leader_election.cc:343] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:37.654848 9618 leader_election.cc:304] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:37.656526 11181 raft_consensus.cc:2749] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:37.660764 11155 tablet_loader.cc:96] loaded metadata for tablet 03e1d998c4314db3b3682a0510393749 (table default.Table_f00a5967b1b7422b830dcf11db2acbd8 [id=6a34d9af17c84984b479bbd10025fba5])
I20260504 13:56:37.662204 11155 tablet_loader.cc:96] loaded metadata for tablet 03fe4ce4f52f48b7b7a804c269e0fd4e (table default.Table_963c93d5d8ed49e6a2052fa1e652da91 [id=399a65bb2fa6428296341de74e964518])
I20260504 13:56:37.666321 11155 tablet_loader.cc:96] loaded metadata for tablet 0be795f04dd54cf19869c937b91d0ddf (table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f])
W20260504 13:56:37.669718 9618 leader_election.cc:343] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:37.670145 11155 tablet_loader.cc:96] loaded metadata for tablet 184fefa651fc411fb5107cc923c02535 (table default.Table_f8a23a8252f84cacb13b338f9d9dc9fc [id=39be2c4d16aa41c0832bfeb23269243a])
I20260504 13:56:37.670200 9618 leader_election.cc:304] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:37.671038 11234 raft_consensus.cc:2749] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:37.672713 11155 tablet_loader.cc:96] loaded metadata for tablet 1c12ffa802b840b1937f6f796083c478 (table default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1])
I20260504 13:56:37.674989 11155 tablet_loader.cc:96] loaded metadata for tablet 1fbe09b4078c4eebb52d7d00bc87371c (table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee])
I20260504 13:56:37.684902 11155 tablet_loader.cc:96] loaded metadata for tablet 23ff0d52a90c4056880190dbd936cb8e (table default.Table_24398dd8ee63413ebb585587353ea155 [id=1d2bbdaf6382433cb8c2a55f62f0388d])
I20260504 13:56:37.686685 11155 tablet_loader.cc:96] loaded metadata for tablet 274bb204041c4e1085395c7bd938aaf1 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5])
I20260504 13:56:37.687471 9932 tablet_bootstrap.cc:492] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:37.687883 9932 ts_tablet_manager.cc:1403] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.007s
W20260504 13:56:37.687829 9331 leader_election.cc:336] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 806407748d564439807fe5d73db9abda (127.8.238.254:36145): Timed out: RequestConsensusVote RPC to 127.8.238.254:36145 timed out after 0.616s (SENT)
I20260504 13:56:37.688203 9331 leader_election.cc:304] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec9f4c3a43d84fc9a420dd10723c01a1; no voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda
I20260504 13:56:37.690291 11155 tablet_loader.cc:96] loaded metadata for tablet 2a629c2a69084b6b98984c8d374eeb1e (table default.Table_a5fa0a0bead94aa3b02d6175fefde171 [id=e9fbd43d30ae4d1dbeea534025125b91])
I20260504 13:56:37.690358 9932 raft_consensus.cc:359] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.690968 9932 raft_consensus.cc:385] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:37.691243 9932 raft_consensus.cc:740] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:37.692427 11155 tablet_loader.cc:96] loaded metadata for tablet 2c09e9076214405eaa8846fe08158bb3 (table default.Table_6e41cb79b95848ba8c7b1f05f961c7f3 [id=b91fe7c5c3844c109756842f93098699])
I20260504 13:56:37.693637 11155 tablet_loader.cc:96] loaded metadata for tablet 2ddf890c17964e7f90409ebe3c4034ef (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e])
I20260504 13:56:37.694787 11155 tablet_loader.cc:96] loaded metadata for tablet 2f6b3e969760442283c3191e062c1839 (table default.Table_5bc32a610edd49ab9641c3726f53f5c6 [id=be5c761518674e7782bfff73366b1675])
I20260504 13:56:37.695935 11155 tablet_loader.cc:96] loaded metadata for tablet 3404ba04d3ec4a44a5e625536b941b58 (table default.Table_7db59fa280b6442b99cf24677b57df83 [id=fdb915ebe83443fd8d2f7f287385f6a4])
I20260504 13:56:37.698418 11155 tablet_loader.cc:96] loaded metadata for tablet 36c8e9ba29a1466082a3e8bedec87b91 (table default.Table_1665799e268a496d9ffc5568fa34a763 [id=403cbbc8a1304e5695a89d3697932c26])
I20260504 13:56:37.699539 11155 tablet_loader.cc:96] loaded metadata for tablet 37b1664312d347288680d97b62c2aecb (table default.Table_861ff05bb9e144ea85a3fb7d75f68749 [id=c4b5a9431b8b4d33b41fa27a357e1f46])
I20260504 13:56:37.697395 9932 consensus_queue.cc:260] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.703562 11155 tablet_loader.cc:96] loaded metadata for tablet 3d9bd2722d23420ea0eac7664357a60c (table default.Table_821e9f466c974a2fbf24c07f8c266e62 [id=469bd5cb4e7f4a8a9c7af947a834ca7b])
I20260504 13:56:37.707994 11155 tablet_loader.cc:96] loaded metadata for tablet 4a1a77ad56634e068401a2bfbc977761 (table default.Table_9bc232a8e4dd4df1a36b35d09dd50d0a [id=66faab34fb324213a874ea6fae3c96a1])
I20260504 13:56:37.709471 11155 tablet_loader.cc:96] loaded metadata for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38])
I20260504 13:56:37.710533 11155 tablet_loader.cc:96] loaded metadata for tablet 4cf1504f0202481ba8b08f9bec5bc3d9 (table default.Table_737cc12f07144a6c92dc2974693a649d [id=e9103e8b725b4e81b6cf094395beef3b])
I20260504 13:56:37.711774 11155 tablet_loader.cc:96] loaded metadata for tablet 4fae9274bb8240798f85b79504a227b3 (table default.Table_e116f9abbb23419d9a3365f7d0e17db9 [id=bee8acaa102249578514a3d4e5f876d3])
I20260504 13:56:37.712890 11155 tablet_loader.cc:96] loaded metadata for tablet 4fdc09a498be470ab04563c62caf7755 (table default.Table_49715b69b840473b9f7b0fdabcc0084e [id=700b914cd7ec44e791cd0a27d4b9bf81])
I20260504 13:56:37.714784 11155 tablet_loader.cc:96] loaded metadata for tablet 5254b1061e2b424487f5d309baaee6b6 (table default.Table_798d2540b8f4489fb11763ccaa36bced [id=049b4fa786fc4dbb8e52931dcaaab90f])
I20260504 13:56:37.716251 9931 raft_consensus.cc:359] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.716763 11155 tablet_loader.cc:96] loaded metadata for tablet 538c8c2412104c3e8b19394a1cc38818 (table default.Table_7dbd11258f34498e82a86f7e3c72f470 [id=7bf52f9bd2a042e59fc4ff612d9846e9])
I20260504 13:56:37.716961 9931 raft_consensus.cc:385] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:37.717743 9931 raft_consensus.cc:740] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:37.718713 11155 tablet_loader.cc:96] loaded metadata for tablet 564d0fafb49a4b4bb3bdb240c74c3410 (table default.Table_1c9b2e63142440a5a45018d187a6cded [id=8a427e40b3374dafa140c9156edcc660])
I20260504 13:56:37.718385 9931 consensus_queue.cc:260] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.719913 11155 tablet_loader.cc:96] loaded metadata for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537])
I20260504 13:56:37.721218 11155 tablet_loader.cc:96] loaded metadata for tablet 59f9115b286d44f0b7eac7c74275cb21 (table default.Table_5944297f8c104978bbc6913ecd8f47a4 [id=3f11131cece44f3ebf3cd0f1e69139ae])
I20260504 13:56:37.722376 11155 tablet_loader.cc:96] loaded metadata for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4])
I20260504 13:56:37.725672 11155 tablet_loader.cc:96] loaded metadata for tablet 6323e1c10e7643f8a28f04e285f620e9 (table default.Table_999e37d0dab54ca68c0fa66f74582109 [id=e37fd7e36e604a60832acf303f0ecb7f])
I20260504 13:56:37.727888 11155 tablet_loader.cc:96] loaded metadata for tablet 644170f17f1b45a19677d958b9745e16 (table default.Table_09fd612a3c4a4303b8f05530e7d49819 [id=eda14d7d5f7d4391a8c470677ffab4ea])
I20260504 13:56:37.729328 9932 ts_tablet_manager.cc:1434] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
I20260504 13:56:37.729875 11155 tablet_loader.cc:96] loaded metadata for tablet 6677a93d01d54f15a506ae1474bcb501 (table default.Table_4842f50414d34bdb8a10802bc059b421 [id=7fc23620ef7342eea52bcce264eb12b3])
I20260504 13:56:37.659492 11026 consensus_queue.cc:1048] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:37.731875 11155 tablet_loader.cc:96] loaded metadata for tablet 68e92cc3da9b4bcfadb6fa26245f1c99 (table default.Table_18f9a8030cea4cc0abedb81cd5accc59 [id=5c4f226dc33b428e92ca77959e075d7c])
I20260504 13:56:37.733206 11234 raft_consensus.cc:493] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:37.733649 11234 raft_consensus.cc:515] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.734124 9932 tablet_bootstrap.cc:492] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:37.668895 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 52620.4us
I20260504 13:56:37.687191 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92480654807d495b879576b0195871bd" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:37.735553 11234 leader_election.cc:290] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:37.688194 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89edd71cf5aa4a6abe2084695d3ed807" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:37.736408 9474 consensus_peers.cc:597] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:37.737498 11155 tablet_loader.cc:96] loaded metadata for tablet 6dd139c347f7406aac340cc6ac994273 (table default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56])
I20260504 13:56:37.739406 11155 tablet_loader.cc:96] loaded metadata for tablet 6f4c8ffb2175493a981c18676485ac04 (table default.Table_93e03451ea09440d847c57c026230518 [id=6ad5facbe9ca40eeb7746baef5c566a7])
I20260504 13:56:37.742216 11155 tablet_loader.cc:96] loaded metadata for tablet 75a43e11c8b34bd7bcddbf6e9b8f30dc (table default.Table_09255abca80f41dc895d81ccd89b9c7a [id=24485cf365ad4f9b89fa0f716fdb34cd])
I20260504 13:56:37.743403 11155 tablet_loader.cc:96] loaded metadata for tablet 7989338bc4bf4bad8fb8fe37587d0a16 (table default.Table_10e92469b9cb4e9b965d0a10cfeecca8 [id=99da5017cead4ddeaa3441064b1ed716])
I20260504 13:56:37.744719 11155 tablet_loader.cc:96] loaded metadata for tablet 7a89bac060214d30924398007dfa1fa1 (table default.Table_2f0fc59ee89c402cb8a0fc23341636e6 [id=00b5922717934e15993bafde9f2513e7])
I20260504 13:56:37.746656 11155 tablet_loader.cc:96] loaded metadata for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13])
I20260504 13:56:37.747929 11155 tablet_loader.cc:96] loaded metadata for tablet 7d27b04d3f4443c5ade9616d8626d50e (table default.Table_8ccb9fc27b374611b20008514d3ad483 [id=6c295b6b72c94ca0b5f1805a555e7f77])
I20260504 13:56:37.749301 11155 tablet_loader.cc:96] loaded metadata for tablet 7eaa40eb4b1c425a9edfb7eaba953b65 (table default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e])
I20260504 13:56:37.749812 11236 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 13:56:37.749931 9530 raft_consensus.cc:2468] T 89edd71cf5aa4a6abe2084695d3ed807 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:37.750739 11155 tablet_loader.cc:96] loaded metadata for tablet 7ed3ee9a13e84594905db47aa33040c2 (table default.Table_6462cb42c34a4eb8976d8d2a4f82d434 [id=6e3151859b6d4da3a477367687862671])
W20260504 13:56:37.750783 9331 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.254:36145, user_credentials={real_user=slave}} blocked reactor thread for 62977.9us
I20260504 13:56:37.752552 11155 tablet_loader.cc:96] loaded metadata for tablet 8702ce3ff1a845f3b70012855cd17efd (table default.Table_24dab2dd1f45402786d80cba80bbad82 [id=80ca58df22214450b894afea377e9c16])
I20260504 13:56:37.753631 11155 tablet_loader.cc:96] loaded metadata for tablet 88619eaa917d4c33a910fc1347ab126e (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c])
I20260504 13:56:37.755304 10013 ts_tablet_manager.cc:1434] T 89edd71cf5aa4a6abe2084695d3ed807 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.220s user 0.006s sys 0.000s
I20260504 13:56:37.755442 11155 tablet_loader.cc:96] loaded metadata for tablet 89edd71cf5aa4a6abe2084695d3ed807 (table default.Table_69c9551ae2cf4ef399e40017a33bad73 [id=2d9b8dca85e9477eb77877d5618e9b6f])
I20260504 13:56:37.756637 11155 tablet_loader.cc:96] loaded metadata for tablet 8a985338790e4887af38c49ba013417f (table default.Table_5636294717ff4c63939c2cf9facfc788 [id=6acf451855024161b7826094f8387a9c])
I20260504 13:56:37.758858 11155 tablet_loader.cc:96] loaded metadata for tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3 (table default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142])
I20260504 13:56:37.760778 11155 tablet_loader.cc:96] loaded metadata for tablet 8d6db45d2e5c41a3b30cede3301694d3 (table default.Table_862ae682dfb44737a72e7c5df628a957 [id=bbef146162cb4c30a3ba42ca2a36fe9f])
I20260504 13:56:37.761996 11155 tablet_loader.cc:96] loaded metadata for tablet 8db7c4732fa048f9950248f45d144c57 (table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62])
W20260504 13:56:37.763648 9618 leader_election.cc:343] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:37.763851 11155 tablet_loader.cc:96] loaded metadata for tablet 9077fb9bb1544a2cbeb96f0d11c400db (table default.Table_06ecf48a04d94ac8a7e954ff2c260dbf [id=e11be55403d24c8bb35f4708da2e9a78])
I20260504 13:56:37.765578 11155 tablet_loader.cc:96] loaded metadata for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1])
I20260504 13:56:37.771417 11234 raft_consensus.cc:493] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:37.771840 11234 raft_consensus.cc:515] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.773524 11234 leader_election.cc:290] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:37.775570 9618 leader_election.cc:304] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:37.776297 11234 raft_consensus.cc:2804] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:37.776630 11234 raft_consensus.cc:493] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:37.776917 11234 raft_consensus.cc:3060] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:37.778911 11155 tablet_loader.cc:96] loaded metadata for tablet 975fa1a4975242a388c15fe73abea1c7 (table default.Table_a660204a2583425aa3ecd09c5cc01368 [id=4683a87561f14c049c1ed8eb68dc24d1])
I20260504 13:56:37.767758 9932 tablet_bootstrap.cc:654] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:37.780004 11155 tablet_loader.cc:96] loaded metadata for tablet 982f7c53a1f0482e862b53aeeee7cd25 (table default.Table_1e6b4c9e684e45b3b2510d3979c1b5c4 [id=ecfa480ab0ea40008ff4507ed6c5b964])
I20260504 13:56:37.781945 11155 tablet_loader.cc:96] loaded metadata for tablet 99e819494171485e9af8bc8ec33a6ff0 (table default.Table_89c38f9ba46e48fab397a7f3d5d1b805 [id=e4f6621892874f979d1c579d487cba44])
I20260504 13:56:37.783107 11155 tablet_loader.cc:96] loaded metadata for tablet 99fe335b131b45a587ec1cf526dd7f79 (table default.Table_655e2e8cf3544fbeb0ce33829bb2c8a0 [id=9db1e031520d405ea4989e30d5194911])
I20260504 13:56:37.786270 11155 tablet_loader.cc:96] loaded metadata for tablet 9f3905bfb30e4b03a7fcf0a2f57e2117 (table default.Table_597b4ca3e37849d0aa9c86eed64d0614 [id=5a597e1cae1d4126ae355289aef1ced6])
I20260504 13:56:37.787434 11155 tablet_loader.cc:96] loaded metadata for tablet 9f80ce19e8f9498b98952a371476f185 (table default.Table_2bed5ea8f43149e8bc3b21cd9d1b3846 [id=15994ccc978545058e324152f70c8c3e])
I20260504 13:56:37.788201 11234 raft_consensus.cc:515] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.789186 11155 tablet_loader.cc:96] loaded metadata for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81])
I20260504 13:56:37.790298 11155 tablet_loader.cc:96] loaded metadata for tablet a26c6a2ffb8746a3941a56c7259412e3 (table default.Table_225ad80b8318432b88a729064c88e8ec [id=a3cca003025745cebf8e7d403d0778cf])
I20260504 13:56:37.791388 11155 tablet_loader.cc:96] loaded metadata for tablet a2bfb681e6824e9f9bdcbea23061d57a (table default.Table_e55147137c794be4bff21afce5d36bbc [id=b45733fe3c5547bea00e3967be81ca35])
I20260504 13:56:37.793171 11155 tablet_loader.cc:96] loaded metadata for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f])
I20260504 13:56:37.794852 11155 tablet_loader.cc:96] loaded metadata for tablet ab4c185332a24d96bc3d0fddc0d20693 (table default.Table_af058f50341e4f478c970f8cb52bf0d7 [id=57ccd7218d454c8c908fe06b798bd83b])
I20260504 13:56:37.795933 11155 tablet_loader.cc:96] loaded metadata for tablet acd5f7d3f7c545d18fe19b64e4bc1de6 (table default.Table_f7b06aadc0024855b08b56cad8c7b6a3 [id=4efd9dd69847446d9f05700d3bd3f914])
I20260504 13:56:37.796986 11155 tablet_loader.cc:96] loaded metadata for tablet af5c5a6cae4e4b6ea3eece2e21a587ec (table default.Table_cdee25533fe54da0a74b421714b0d7c4 [id=81df524d85774544807ba1902be1aae1])
I20260504 13:56:37.799623 11155 tablet_loader.cc:96] loaded metadata for tablet b13303d2a14c4187b82da9f25ce9753e (table default.Table_72e3a254561b4cae927e6b6bcacd87b6 [id=f227576f105c4b1fbd459b3638ce684b])
I20260504 13:56:37.800716 11155 tablet_loader.cc:96] loaded metadata for tablet b320a37b29844c7593a12000eaa3da43 (table default.Table_2099388b6e2a455295be76a6d072758e [id=63e34a66cd644ab7bede27373e321714])
I20260504 13:56:37.802454 11155 tablet_loader.cc:96] loaded metadata for tablet b37a738f2902402ca76e4edebcdba474 (table default.Table_91cc6c38c9ff43dca25814664068b1c6 [id=da5c427e75e74ea1ae3a5961ed27e0de])
I20260504 13:56:37.804613 11155 tablet_loader.cc:96] loaded metadata for tablet b6103d953ace4cb8be3c580b3316f87c (table default.Table_a992fb6e038e4408a781d6d8ead37f54 [id=937a02c37dbd45d5bc92979ceaee1c06])
I20260504 13:56:37.805533 11234 leader_election.cc:290] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:37.806411 11155 tablet_loader.cc:96] loaded metadata for tablet b7b942037c0e45a6b3cccba288b90b00 (table default.Table_dd11ea89ad7248c39ec114af58cf598f [id=14bd1983ba95468e9e70d8f5724f42e6])
I20260504 13:56:37.807492 11155 tablet_loader.cc:96] loaded metadata for tablet b84c3bedc7a84e0d8a873fb15689b314 (table default.Table_7f912e1d4381428e9572cbb4b13f4eec [id=fb18c55d21704b2ab04dbf35d21377bf])
I20260504 13:56:37.801002 9932 tablet_bootstrap.cc:492] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:37.807785 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e819494171485e9af8bc8ec33a6ff0" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:37.808516 11155 tablet_loader.cc:96] loaded metadata for tablet b8e4239942f34fb081a026a35740e465 (table default.Table_11d8ab61a6044154bb599875a0bc9004 [id=73e12524eb9e416782bf9f7fb58c93bd])
I20260504 13:56:37.809278 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59f9115b286d44f0b7eac7c74275cb21" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:37.809549 11155 tablet_loader.cc:96] loaded metadata for tablet b8ffeae4e5af4cc8a3eb3bc0bedf18cc (table default.Table_003cbc6fdf0b4dddaba008af167dc621 [id=4629003a2d8d461c9e309d49ddecea6a])
I20260504 13:56:37.810113 9932 ts_tablet_manager.cc:1403] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.078s user 0.013s sys 0.000s
I20260504 13:56:37.810578 11155 tablet_loader.cc:96] loaded metadata for tablet ba4584755aae47e0b0f14c6c55474d18 (table default.Table_2d5a582adb634895a55193dd861f3b60 [id=30f349197d564e8da82855fec17b4cd2])
I20260504 13:56:37.810631 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89edd71cf5aa4a6abe2084695d3ed807" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:37.811123 9530 raft_consensus.cc:3060] T 89edd71cf5aa4a6abe2084695d3ed807 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:37.811685 11155 tablet_loader.cc:96] loaded metadata for tablet bae90898739b4303af0608e39e8c0367 (table default.Table_073a6dae9da444bea1f96b2f33fe1a85 [id=39c692c399dd43ca963357b6818871cf])
I20260504 13:56:37.813472 11155 tablet_loader.cc:96] loaded metadata for tablet bd635a863f7a41d087947f619e41f95c (table default.Table_849bbe7902ca44969c5208828774fa06 [id=c146e84e0e844deda9af40562f980047])
I20260504 13:56:37.815208 11155 tablet_loader.cc:96] loaded metadata for tablet bdf1c3cef3334a849f7b5e429b7f4990 (table default.Table_42f086bc327c42ccad96925c681a386a [id=2c6aa17153154189a33f5f3aa00b62e9])
I20260504 13:56:37.816272 11155 tablet_loader.cc:96] loaded metadata for tablet c1a01231bb044433ae5827d07a6592f8 (table default.Table_3387191fcd6046abbf257447bfbb27f7 [id=d195f5bef6ce49f1b87a4bf5f516eebb])
W20260504 13:56:37.816267 9618 leader_election.cc:343] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:37.817324 11155 tablet_loader.cc:96] loaded metadata for tablet c3112c3c3629412e9b76407c7cba8f92 (table default.Table_b59e5e285d50401e94d51afeb4cbc500 [id=8f16af9aec63405cbae2a91b84fa3009])
I20260504 13:56:37.819823 11155 tablet_loader.cc:96] loaded metadata for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a])
I20260504 13:56:37.821595 11155 tablet_loader.cc:96] loaded metadata for tablet c6ebc96408d64c2da7dc6c4fc158995c (table default.Table_5ff6dab0e47d41e681533fca02a36153 [id=230a8885dd2b4995815b650d3d9f9f75])
I20260504 13:56:37.819497 9932 raft_consensus.cc:359] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.822650 11155 tablet_loader.cc:96] loaded metadata for tablet c8989cdde4ca4e349079596e33723b2c (table default.Table_2d928cb3553f40bc8e825d0741ef3199 [id=51b254be98ee457d892039784e4a5f0b])
I20260504 13:56:37.822762 9530 raft_consensus.cc:2468] T 89edd71cf5aa4a6abe2084695d3ed807 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:37.823803 11155 tablet_loader.cc:96] loaded metadata for tablet cc2f95dadd544f57bd6d1467d30e954d (table default.Table_2585508fc4214c8b82d2957644309ac2 [id=256b605e5a2342ed883ec30e3de0ffda])
I20260504 13:56:37.823351 9932 raft_consensus.cc:385] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:37.825582 11155 tablet_loader.cc:96] loaded metadata for tablet cf11f17e95dc411294c755e17c57811e (table default.Table_2dc0381bd6c84ab59d5bb422de89c0e2 [id=ed756f69850642188b881bd3c680a622])
I20260504 13:56:37.824754 9932 raft_consensus.cc:740] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:37.829069 11155 tablet_loader.cc:96] loaded metadata for tablet dd3c045379734134b49e94340679c11d (table default.Table_443402c40ae5443daf4ba4258b847303 [id=98126e5c27cd4d3da57fcd88d481de79])
I20260504 13:56:37.830108 11155 tablet_loader.cc:96] loaded metadata for tablet ddd30e9259a04d59b5c1bb82aabf7952 (table default.Table_512ae07c2df74cb99bb77006d985f2da [id=09959f81c69c49778a6ee33dc1e85aae])
I20260504 13:56:37.828464 9932 consensus_queue.cc:260] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.838299 9932 ts_tablet_manager.cc:1434] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.028s user 0.005s sys 0.002s
I20260504 13:56:37.832643 11155 tablet_loader.cc:96] loaded metadata for tablet de6714ee86d64112baf13dd1b2b87582 (table default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469])
I20260504 13:56:37.843302 11155 tablet_loader.cc:96] loaded metadata for tablet df71dc4f2cf84cda8cfc534a0eea86f8 (table default.Table_1611aa9f4dba40e09dae4433e2d20154 [id=2fb2c56e2fb44b9aac476e06e36e5778])
I20260504 13:56:37.844412 11155 tablet_loader.cc:96] loaded metadata for tablet dffddd4b89984c54bc26dcb4a9488525 (table default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568])
I20260504 13:56:37.845436 11155 tablet_loader.cc:96] loaded metadata for tablet e0b9e47b0a4443c5830422b9d0553446 (table default.Table_4207c9e8019c4088b934376c5568d101 [id=3ca89308de104a5fa27cac4c62047304])
I20260504 13:56:37.846455 11155 tablet_loader.cc:96] loaded metadata for tablet e0e3b36e1f654a3a8df51480801dabd7 (table default.Table_9e9ef3353dcf4b1fadaa2f51bc8f7041 [id=8dcabd7d67334598926f927a62e630d6])
I20260504 13:56:37.845695 9932 tablet_bootstrap.cc:492] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:37.847638 11155 tablet_loader.cc:96] loaded metadata for tablet e0e70772b7ce4f27a941918aaf66a865 (table default.Table_60fe6ee7a2414e2aa8ea2d57ee4a827d [id=1de44a1bbf06492084cfd70cf540b213])
I20260504 13:56:37.849299 11155 tablet_loader.cc:96] loaded metadata for tablet e29a7365caf04506b15089f10e6c1677 (table default.Table_6b0124eccb02454b9195b8ddaae5bd38 [id=d47a2c6a9cb34ba4ade10fc23913bf7d])
I20260504 13:56:37.850356 11155 tablet_loader.cc:96] loaded metadata for tablet e53381ee0e7f4e9781b833b435bca831 (table default.Table_9b40ba316993413181a9743fb7bab7fb [id=fb4e5ce5526e44de8effdd766f4c257f])
I20260504 13:56:37.851418 11155 tablet_loader.cc:96] loaded metadata for tablet e5d63f665166466dba1e7e36c5e19506 (table default.Table_363d6634e42e4dd6a5686c1f0e401ef2 [id=d0d8b54fb69540589420b87f90b4c823])
I20260504 13:56:37.852384 11155 tablet_loader.cc:96] loaded metadata for tablet e9e66dacf5794b6195575556be889bce (table default.Table_d6a5a22d78264e799e80d31d5749600a [id=d63acf266ca440729128a717eed32433])
W20260504 13:56:37.852460 9618 leader_election.cc:343] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:37.853770 9618 leader_election.cc:304] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:37.854442 11187 raft_consensus.cc:2804] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:37.854885 11155 tablet_loader.cc:96] loaded metadata for tablet eef975650610423c88ce8e4cfb0b013b (table default.Table_0134ffe7086440ccbcadd076d8fe5d01 [id=693e9b6ed89b4ab7b9d5dc7402e17f20])
I20260504 13:56:37.854885 11187 raft_consensus.cc:697] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:37.855722 11187 consensus_queue.cc:237] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.856433 11155 tablet_loader.cc:96] loaded metadata for tablet f337e9698fe64282a684e90d9c61da5a (table default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12])
I20260504 13:56:37.857533 11155 tablet_loader.cc:96] loaded metadata for tablet f3e86027e3e74bc196e9a9ffc9fdf8a5 (table default.Table_780e641042db4fd5aeb8af8b1aa08c72 [id=6cb26719d5fe45ccbdef18571c902228])
I20260504 13:56:37.861325 10013 tablet_bootstrap.cc:492] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:37.858625 11155 tablet_loader.cc:96] loaded metadata for tablet f71f444ce9214e3f98eff6ac8ca7dde7 (table default.Table_df8874a20e834872a12300a4f5628273 [id=7fc0413d79184a2dbe5c470b8506b4e1])
I20260504 13:56:37.863101 11155 tablet_loader.cc:96] loaded metadata for tablet fac3e36cfa4848a1ab015f89c56c2740 (table default.Table_8cae213b77994b9fba63b8a7c5f27249 [id=df86bccb4ca8414c90f9fd8753724d89])
I20260504 13:56:37.864240 11155 tablet_loader.cc:96] loaded metadata for tablet fe7ab3b577b74a9c9a40cb15c94f8aa1 (table default.Table_dee1335eff89438e9c68835a4c258031 [id=1714df2b31ff4865b6b357500d121a26])
I20260504 13:56:37.868470 10956 consensus_queue.cc:1048] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:37.870010 11155 catalog_manager.cc:1486] Time spent T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: Loading table and tablet metadata into memory: real 1.963s user 0.377s sys 0.044s
I20260504 13:56:37.870249 11155 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260504 13:56:37.870167 9199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
W20260504 13:56:37.870965 9199 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35460 (request call id 534) took 800 ms (client timeout 616 ms). Trace:
I20260504 13:56:37.871256 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
W20260504 13:56:37.871371 9199 rpcz_store.cc:269] 0504 13:56:37.070727 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:37.070918 (+ 191us) service_pool.cc:224] Handling call
0504 13:56:37.870879 (+799961us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 13:56:37.872069 9198 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35460 (request call id 533) took 1475 ms (client timeout 616 ms). Trace:
W20260504 13:56:37.872736 9198 rpcz_store.cc:269] 0504 13:56:36.396623 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:36.396748 (+ 125us) service_pool.cc:224] Handling call
0504 13:56:37.871988 (+1475240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:37.903796 9932 tablet_bootstrap.cc:654] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:37.918067 10013 tablet_bootstrap.cc:654] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:37.930703 11155 catalog_manager.cc:1269] Loaded cluster ID: 302d3e241578494980ac3509cf5306fb
I20260504 13:56:37.933256 11155 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260504 13:56:37.986150 11187 raft_consensus.cc:493] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:37.986642 11187 raft_consensus.cc:515] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:37.988581 11187 leader_election.cc:290] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:37.991186 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6103d953ace4cb8be3c580b3316f87c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:37.993198 9618 leader_election.cc:343] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.005650 11155 catalog_manager.cc:1514] Loading token signing keys...
I20260504 13:56:38.010214 11244 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 806407748d564439807fe5d73db9abda)
I20260504 13:56:38.026919 9932 tablet_bootstrap.cc:492] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:38.027359 9932 ts_tablet_manager.cc:1403] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.187s user 0.010s sys 0.034s
I20260504 13:56:38.031198 9932 raft_consensus.cc:359] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.031843 9932 raft_consensus.cc:385] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:38.032095 9932 raft_consensus.cc:740] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:38.032632 9932 consensus_queue.cc:260] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.034297 9932 ts_tablet_manager.cc:1434] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260504 13:56:38.035096 9932 tablet_bootstrap.cc:492] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:38.039228 10013 tablet_bootstrap.cc:492] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:38.039659 10013 ts_tablet_manager.cc:1403] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.179s user 0.010s sys 0.000s
I20260504 13:56:38.010730 11244 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:38.046079 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92480654807d495b879576b0195871bd" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:38.046842 11244 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:38.055419 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 5 candidate_status { last_received { term: 4 index: 355 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:38.059484 11155 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: Loaded TSK: 0
I20260504 13:56:38.064021 11155 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260504 13:56:38.073135 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:38.093487 11187 raft_consensus.cc:493] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:38.093972 11187 raft_consensus.cc:515] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.096042 11187 leader_election.cc:290] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:38.097602 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da61c91e10264fa49fe397d9a9b014bb" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:38.099283 9618 leader_election.cc:343] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.101748 9932 tablet_bootstrap.cc:654] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:38.104507 9931 ts_tablet_manager.cc:1434] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.474s user 0.004s sys 0.001s
I20260504 13:56:38.047683 10013 raft_consensus.cc:359] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.105836 10013 raft_consensus.cc:385] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:38.106173 10013 raft_consensus.cc:740] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:38.106596 9931 tablet_bootstrap.cc:492] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:38.107010 10013 consensus_queue.cc:260] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.109635 10013 ts_tablet_manager.cc:1434] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.067s user 0.005s sys 0.000s
I20260504 13:56:38.188704 10956 consensus_queue.cc:1048] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:38.192905 11026 raft_consensus.cc:493] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:38.193397 11026 raft_consensus.cc:515] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.196025 11026 leader_election.cc:290] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:38.210533 9328 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet a862a43c5ac443d9bac98e0472ca23b2 consensus state has no leader
I20260504 13:56:38.212612 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "538c8c2412104c3e8b19394a1cc38818" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:38.213181 9681 raft_consensus.cc:2468] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:38.214336 9473 leader_election.cc:304] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:38.214774 10013 tablet_bootstrap.cc:492] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:38.215488 11233 raft_consensus.cc:2804] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:38.215847 11233 raft_consensus.cc:493] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:38.216145 11233 raft_consensus.cc:3060] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:38.224655 10013 tablet_bootstrap.cc:654] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:38.245194 10013 tablet_bootstrap.cc:492] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:38.247123 10013 ts_tablet_manager.cc:1403] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.000s
I20260504 13:56:38.262451 10013 raft_consensus.cc:359] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.280524 9932 tablet_bootstrap.cc:492] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:38.282459 9186 catalog_manager.cc:5671] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:38.295032 11187 raft_consensus.cc:493] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:38.297550 11187 raft_consensus.cc:515] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.304628 11187 leader_election.cc:290] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:38.307032 9932 ts_tablet_manager.cc:1403] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.272s user 0.013s sys 0.005s
I20260504 13:56:38.309651 9932 raft_consensus.cc:359] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.310263 9932 raft_consensus.cc:385] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:38.310527 9932 raft_consensus.cc:740] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:38.292972 11233 raft_consensus.cc:515] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.306365 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21d6358b451d4b8d87a22e30e0780f44" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:38.325383 11233 leader_election.cc:290] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:38.328356 9826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "538c8c2412104c3e8b19394a1cc38818" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:38.338774 10013 raft_consensus.cc:385] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:38.339105 10013 raft_consensus.cc:740] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:38.339844 10013 consensus_queue.cc:260] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.341115 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "538c8c2412104c3e8b19394a1cc38818" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:38.342239 10013 ts_tablet_manager.cc:1434] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.093s user 0.005s sys 0.000s
W20260504 13:56:38.324981 9618 leader_election.cc:343] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.311141 9932 consensus_queue.cc:260] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.360560 11187 raft_consensus.cc:493] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:38.361012 11187 raft_consensus.cc:515] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.367964 11187 leader_election.cc:290] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:38.369678 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99fe335b131b45a587ec1cf526dd7f79" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:38.371140 9618 leader_election.cc:343] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.375371 9931 tablet_bootstrap.cc:654] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:38.390336 9932 ts_tablet_manager.cc:1434] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.083s user 0.006s sys 0.000s
I20260504 13:56:38.413717 11187 raft_consensus.cc:493] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:38.414359 11187 raft_consensus.cc:515] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.415400 10013 tablet_bootstrap.cc:492] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:38.416559 11187 leader_election.cc:290] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:38.417600 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75a43e11c8b34bd7bcddbf6e9b8f30dc" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:38.418797 9618 leader_election.cc:343] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.428499 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "538c8c2412104c3e8b19394a1cc38818" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:38.429014 9681 raft_consensus.cc:3060] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:38.438154 9681 raft_consensus.cc:2468] T 538c8c2412104c3e8b19394a1cc38818 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:38.476670 11199 raft_consensus.cc:493] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:38.477293 11199 raft_consensus.cc:515] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.479621 11199 leader_election.cc:290] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:38.509182 9932 tablet_bootstrap.cc:492] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:38.516440 9474 consensus_peers.cc:597] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:38.517202 9474 consensus_peers.cc:597] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:38.520582 11260 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:38.521230 11260 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:38.524461 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
W20260504 13:56:38.525013 9474 leader_election.cc:343] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:38.526094 9474 leader_election.cc:343] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.537674 9473 leader_election.cc:304] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:38.538785 10956 raft_consensus.cc:2804] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
W20260504 13:56:38.551981 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:38.570276 10956 raft_consensus.cc:697] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:38.571115 10956 consensus_queue.cc:237] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.576848 10013 tablet_bootstrap.cc:700] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00: Time spent opening tablet: real 0.154s user 0.004s sys 0.000s
I20260504 13:56:38.571866 11233 consensus_queue.cc:1048] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.040s
I20260504 13:56:38.584784 10013 tablet_bootstrap.cc:654] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:38.586447 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e819494171485e9af8bc8ec33a6ff0" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:38.587747 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59f9115b286d44f0b7eac7c74275cb21" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:38.588855 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89edd71cf5aa4a6abe2084695d3ed807" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:38.589958 9826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89edd71cf5aa4a6abe2084695d3ed807" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:38.591019 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6103d953ace4cb8be3c580b3316f87c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:38.592800 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da61c91e10264fa49fe397d9a9b014bb" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:38.595664 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21d6358b451d4b8d87a22e30e0780f44" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:38.564333 9932 tablet_bootstrap.cc:654] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:38.611578 11199 consensus_queue.cc:1048] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:38.612535 10013 tablet_bootstrap.cc:492] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:38.612959 10013 ts_tablet_manager.cc:1403] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.270s user 0.011s sys 0.000s
I20260504 13:56:38.624539 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99fe335b131b45a587ec1cf526dd7f79" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:38.633874 11010 consensus_queue.cc:1048] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:38.637836 9767 leader_election.cc:343] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Not found: Tablet not found: 41d41f8ba6dc4bb18116cb2b2dfabff0
W20260504 13:56:38.647460 9620 leader_election.cc:343] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.647881 9620 leader_election.cc:304] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:38.648784 11187 raft_consensus.cc:2749] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:38.652042 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75a43e11c8b34bd7bcddbf6e9b8f30dc" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:38.644781 9768 leader_election.cc:343] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:38.653735 9768 leader_election.cc:304] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:38.654592 11252 raft_consensus.cc:2749] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:38.667567 9932 tablet_bootstrap.cc:492] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:38.667953 9932 ts_tablet_manager.cc:1403] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.277s user 0.006s sys 0.005s
I20260504 13:56:38.668918 11187 raft_consensus.cc:493] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:38.669363 11187 raft_consensus.cc:515] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.670130 9932 raft_consensus.cc:359] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:38.670732 9932 raft_consensus.cc:385] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:38.673247 11187 leader_election.cc:290] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:38.674885 9932 raft_consensus.cc:740] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:38.675566 9932 consensus_queue.cc:260] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:38.677752 9932 ts_tablet_manager.cc:1434] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260504 13:56:38.677563 9534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57983fcecdb64a42884f797b4db57df0" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:38.678546 9932 tablet_bootstrap.cc:492] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:38.679076 9618 leader_election.cc:343] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.684294 9932 tablet_bootstrap.cc:654] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:38.685505 10013 raft_consensus.cc:359] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.686161 10013 raft_consensus.cc:385] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:38.686434 10013 raft_consensus.cc:740] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:38.687201 10013 consensus_queue.cc:260] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.688289 11260 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:56:38.696238 10013 ts_tablet_manager.cc:1434] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.083s user 0.005s sys 0.000s
I20260504 13:56:38.697813 10013 tablet_bootstrap.cc:492] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:38.697932 9620 leader_election.cc:343] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.698302 9620 leader_election.cc:304] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:38.699288 11187 raft_consensus.cc:2749] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:38.700274 9620 leader_election.cc:343] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.700697 9620 leader_election.cc:304] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:38.701668 11187 raft_consensus.cc:2749] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:38.703941 10810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:38.705479 9328 leader_election.cc:304] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec9f4c3a43d84fc9a420dd10723c01a1; no voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda
W20260504 13:56:38.714030 10690 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.238.252:44781 timed out after 0.664s (SENT) [suppressed 1 similar messages]
W20260504 13:56:38.718614 9620 leader_election.cc:343] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.720214 11260 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 13:56:38.721591 9620 leader_election.cc:343] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.725889 9931 tablet_bootstrap.cc:492] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:38.726301 9931 ts_tablet_manager.cc:1403] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.620s user 0.010s sys 0.005s
W20260504 13:56:38.733134 9620 leader_election.cc:343] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.733675 9620 leader_election.cc:304] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:38.734443 11187 raft_consensus.cc:2749] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:38.736200 11026 raft_consensus.cc:493] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:38.736670 11026 raft_consensus.cc:515] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:38.738449 9620 leader_election.cc:343] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.738622 11026 leader_election.cc:290] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:38.738862 9620 leader_election.cc:304] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:38.739494 11187 raft_consensus.cc:2749] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:38.742014 9620 leader_election.cc:343] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.742506 9620 leader_election.cc:304] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:38.756736 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a89bac060214d30924398007dfa1fa1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:38.758388 11187 raft_consensus.cc:2749] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:38.759770 9620 leader_election.cc:343] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.760453 9620 leader_election.cc:304] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:38.761512 11187 raft_consensus.cc:2749] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:38.795632 10013 tablet_bootstrap.cc:654] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:38.797408 11187 consensus_queue.cc:1048] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:38.790889 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 45383.2us
I20260504 13:56:38.821311 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57983fcecdb64a42884f797b4db57df0" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:38.790309 9931 raft_consensus.cc:359] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.832211 9931 raft_consensus.cc:385] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:38.832455 9931 raft_consensus.cc:740] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:38.834502 11206 raft_consensus.cc:493] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:38.834933 11206 raft_consensus.cc:515] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:38.836697 11206 leader_election.cc:290] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:38.838755 9620 leader_election.cc:343] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:38.839345 9768 leader_election.cc:343] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 8bb77106b48a45a2aaa938dc474fa8bd
I20260504 13:56:38.841321 9931 consensus_queue.cc:260] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.842756 11187 raft_consensus.cc:493] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:38.843317 11187 raft_consensus.cc:515] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:38.844496 9830 raft_consensus.cc:3060] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:38.845340 11187 leader_election.cc:290] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:38.846633 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88619eaa917d4c33a910fc1347ab126e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:38.847882 9618 leader_election.cc:343] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.852967 9932 tablet_bootstrap.cc:492] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:38.853343 9932 ts_tablet_manager.cc:1403] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.175s user 0.022s sys 0.009s
I20260504 13:56:38.846983 9931 ts_tablet_manager.cc:1434] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.120s user 0.001s sys 0.005s
I20260504 13:56:38.867290 9932 raft_consensus.cc:359] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.863840 9931 tablet_bootstrap.cc:492] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:38.883803 11225 consensus_queue.cc:1048] T 2a629c2a69084b6b98984c8d374eeb1e P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:38.894310 9932 raft_consensus.cc:385] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:38.894629 9932 raft_consensus.cc:740] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:38.895254 9932 consensus_queue.cc:260] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.916554 11187 raft_consensus.cc:493] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:38.917023 11187 raft_consensus.cc:515] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:38.918402 9931 tablet_bootstrap.cc:654] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:38.918998 11187 leader_election.cc:290] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:38.919829 9932 ts_tablet_manager.cc:1434] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.066s user 0.007s sys 0.000s
I20260504 13:56:38.920188 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df71dc4f2cf84cda8cfc534a0eea86f8" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:38.921427 9618 leader_election.cc:343] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:38.925292 9932 tablet_bootstrap.cc:492] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:38.933125 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 150775us
I20260504 13:56:38.842769 9620 leader_election.cc:304] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:38.974449 10013 tablet_bootstrap.cc:492] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:38.980654 11234 raft_consensus.cc:2749] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:38.981611 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 142877us
I20260504 13:56:38.993402 10013 ts_tablet_manager.cc:1403] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.296s user 0.012s sys 0.004s
I20260504 13:56:39.001644 11234 raft_consensus.cc:493] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.002080 11234 raft_consensus.cc:515] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.003901 11234 leader_election.cc:290] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:39.004103 9931 tablet_bootstrap.cc:492] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:39.005411 9931 ts_tablet_manager.cc:1403] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.142s user 0.015s sys 0.005s
I20260504 13:56:39.005591 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e1d998c4314db3b3682a0510393749" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:39.006963 9618 leader_election.cc:343] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.007872 9931 raft_consensus.cc:359] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.008566 9931 raft_consensus.cc:385] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.008831 9931 raft_consensus.cc:740] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.009495 9931 consensus_queue.cc:260] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.010821 9931 ts_tablet_manager.cc:1434] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260504 13:56:39.013535 9474 leader_election.cc:343] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:39.010918 9620 leader_election.cc:343] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.032135 9620 leader_election.cc:304] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:39.032871 11187 raft_consensus.cc:2749] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:39.037125 9932 tablet_bootstrap.cc:654] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:39.050236 9931 tablet_bootstrap.cc:492] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:39.059779 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 48887.3us
I20260504 13:56:39.067379 9678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a89bac060214d30924398007dfa1fa1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:39.068033 9678 raft_consensus.cc:2468] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:39.070619 10013 raft_consensus.cc:359] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.071360 10013 raft_consensus.cc:385] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.071660 10013 raft_consensus.cc:740] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.072268 10013 consensus_queue.cc:260] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.074779 10013 ts_tablet_manager.cc:1434] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.069s user 0.003s sys 0.002s
W20260504 13:56:39.086508 9767 leader_election.cc:343] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Not found: Tablet not found: 8bb77106b48a45a2aaa938dc474fa8bd
I20260504 13:56:39.086907 9767 leader_election.cc:304] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:39.088030 11206 raft_consensus.cc:2749] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:39.089226 9931 tablet_bootstrap.cc:654] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:39.092861 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:39.096895 9932 tablet_bootstrap.cc:492] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:39.097594 9932 ts_tablet_manager.cc:1403] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.173s user 0.015s sys 0.001s
I20260504 13:56:39.101322 9932 raft_consensus.cc:359] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.113140 9932 raft_consensus.cc:385] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.113445 9932 raft_consensus.cc:740] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.114239 9932 consensus_queue.cc:260] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.120164 9932 ts_tablet_manager.cc:1434] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20260504 13:56:39.121644 9932 tablet_bootstrap.cc:492] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:39.122848 9187 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
I20260504 13:56:39.123948 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88619eaa917d4c33a910fc1347ab126e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:39.124559 9826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df71dc4f2cf84cda8cfc534a0eea86f8" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:39.128989 9826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e1d998c4314db3b3682a0510393749" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:39.132571 9881 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
W20260504 13:56:39.137240 9187 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 57983fcecdb64a42884f797b4db57df0 consensus state has no leader
I20260504 13:56:39.137493 9826 raft_consensus.cc:3060] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:39.139349 9187 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:45997 (request call id 266) took 1026 ms (client timeout 1000 ms). Trace:
W20260504 13:56:39.139745 9187 rpcz_store.cc:269] 0504 13:56:38.112728 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:38.181635 (+ 68907us) service_pool.cc:224] Handling call
0504 13:56:39.139289 (+957654us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":173,"mutex_wait_us":451,"reported_tablets":4}
W20260504 13:56:39.141870 9186 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:39697 (request call id 226) took 867 ms (client timeout 1000 ms). Trace:
W20260504 13:56:39.142246 9186 rpcz_store.cc:269] 0504 13:56:38.274383 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:38.274512 (+ 129us) service_pool.cc:224] Handling call
0504 13:56:39.141820 (+867308us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:39.063912 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:39.066608 (+ 2696us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:39.066641 (+ 33us) write_op.cc:435] Acquired schema lock
0504 13:56:39.066658 (+ 17us) tablet.cc:662] Decoding operations
0504 13:56:39.066979 (+ 321us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:39.067039 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:39.067073 (+ 34us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:39.067273 (+ 200us) tablet.cc:701] Row locks acquired
0504 13:56:39.067292 (+ 19us) write_op.cc:260] PREPARE: finished
0504 13:56:39.067438 (+ 146us) write_op.cc:270] Start()
0504 13:56:39.067553 (+ 115us) write_op.cc:276] Timestamp: P: 1777902999067377 usec, L: 0
0504 13:56:39.067597 (+ 44us) op_driver.cc:348] REPLICATION: starting
0504 13:56:39.068232 (+ 635us) log.cc:844] Serialized 533 byte log entry
0504 13:56:39.106022 (+ 37790us) op_driver.cc:464] REPLICATION: finished
0504 13:56:39.119874 (+ 13852us) write_op.cc:301] APPLY: starting
0504 13:56:39.119930 (+ 56us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:39.120015 (+ 85us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:39.120108 (+ 93us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:39.120267 (+ 159us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:39.120416 (+ 149us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:39.120457 (+ 41us) write_op.cc:312] APPLY: finished
0504 13:56:39.120636 (+ 179us) log.cc:844] Serialized 23 byte log entry
0504 13:56:39.120825 (+ 189us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:39.120890 (+ 65us) write_op.cc:454] Released schema lock
0504 13:56:39.120923 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":13667,"num_ops":1,"prepare.queue_time_us":21794,"prepare.run_cpu_time_us":2985,"prepare.run_wall_time_us":38165,"raft.queue_time_us":97,"raft.run_cpu_time_us":729,"raft.run_wall_time_us":1523,"replication_time_us":38330,"thread_start_us":126301,"threads_started":3,"wal-append.queue_time_us":23963}]]}
I20260504 13:56:39.145888 9932 tablet_bootstrap.cc:654] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:39.147071 9588 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:39.172791 10013 tablet_bootstrap.cc:492] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:39.174083 11187 raft_consensus.cc:493] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.174536 11187 raft_consensus.cc:515] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.176503 11187 leader_election.cc:290] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:39.176734 9931 tablet_bootstrap.cc:492] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:39.177119 9931 ts_tablet_manager.cc:1403] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.129s user 0.009s sys 0.000s
I20260504 13:56:39.178859 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1a01231bb044433ae5827d07a6592f8" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:39.180104 9618 leader_election.cc:343] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:39.182893 9175 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 57983fcecdb64a42884f797b4db57df0 consensus state has no leader
I20260504 13:56:39.182500 9931 raft_consensus.cc:359] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:39.183272 9931 raft_consensus.cc:385] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.183540 9931 raft_consensus.cc:740] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.184103 9931 consensus_queue.cc:260] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:39.186271 9931 ts_tablet_manager.cc:1434] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260504 13:56:39.193965 9932 tablet_bootstrap.cc:492] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:39.194752 9932 ts_tablet_manager.cc:1403] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.073s user 0.007s sys 0.006s
I20260504 13:56:39.201735 9932 raft_consensus.cc:359] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.204830 9931 tablet_bootstrap.cc:492] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:39.205241 9932 raft_consensus.cc:385] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.207213 9932 raft_consensus.cc:740] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.218345 10956 raft_consensus.cc:493] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.218812 10956 raft_consensus.cc:515] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.219904 9932 consensus_queue.cc:260] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.220654 10956 leader_election.cc:290] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:39.226625 9932 ts_tablet_manager.cc:1434] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.031s user 0.004s sys 0.002s
I20260504 13:56:39.229452 9931 tablet_bootstrap.cc:654] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:39.246441 9931 tablet_bootstrap.cc:492] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:39.246865 9931 ts_tablet_manager.cc:1403] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.060s user 0.007s sys 0.001s
I20260504 13:56:39.247956 9932 tablet_bootstrap.cc:492] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:39.252395 9931 raft_consensus.cc:359] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.253113 9931 raft_consensus.cc:385] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.253494 9931 raft_consensus.cc:740] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.261133 9473 leader_election.cc:304] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:39.261293 9931 consensus_queue.cc:260] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.262861 11274 raft_consensus.cc:2804] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:39.263175 9931 ts_tablet_manager.cc:1434] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260504 13:56:39.263252 11274 raft_consensus.cc:493] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:39.263643 11274 raft_consensus.cc:3060] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:39.264180 9931 tablet_bootstrap.cc:492] T dffddd4b89984c54bc26dcb4a9488525 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:39.266587 11187 raft_consensus.cc:493] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.267014 11187 raft_consensus.cc:515] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:39.268823 11187 leader_election.cc:290] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:39.270108 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dd139c347f7406aac340cc6ac994273" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:39.271481 9618 leader_election.cc:343] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.274200 10013 tablet_bootstrap.cc:654] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:39.274969 9931 tablet_bootstrap.cc:654] T dffddd4b89984c54bc26dcb4a9488525 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:39.286827 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cb30835150949f39f6e7e7563381763" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:39.292234 9931 tablet_bootstrap.cc:492] T dffddd4b89984c54bc26dcb4a9488525 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:39.292618 9931 ts_tablet_manager.cc:1403] T dffddd4b89984c54bc26dcb4a9488525 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.002s
I20260504 13:56:39.294729 9931 raft_consensus.cc:359] T dffddd4b89984c54bc26dcb4a9488525 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.295302 9931 raft_consensus.cc:385] T dffddd4b89984c54bc26dcb4a9488525 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.295557 9931 raft_consensus.cc:740] T dffddd4b89984c54bc26dcb4a9488525 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.296223 9931 consensus_queue.cc:260] T dffddd4b89984c54bc26dcb4a9488525 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.298488 9931 ts_tablet_manager.cc:1434] T dffddd4b89984c54bc26dcb4a9488525 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260504 13:56:39.299381 9931 tablet_bootstrap.cc:492] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:39.316865 11234 raft_consensus.cc:493] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.317286 11234 raft_consensus.cc:515] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.319084 11234 leader_election.cc:290] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:39.320816 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2bfb681e6824e9f9bdcbea23061d57a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:39.353071 9618 leader_election.cc:343] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.354085 10956 raft_consensus.cc:493] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.354601 10956 raft_consensus.cc:515] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.356922 10956 leader_election.cc:290] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:39.326494 9474 leader_election.cc:343] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.368227 11234 raft_consensus.cc:493] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.368681 11234 raft_consensus.cc:515] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:39.370268 9620 leader_election.cc:343] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.370332 11234 leader_election.cc:290] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:39.370654 9620 leader_election.cc:304] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:39.370800 11234 raft_consensus.cc:493] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.371364 11187 raft_consensus.cc:2749] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:39.371603 9620 leader_election.cc:343] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.371937 9620 leader_election.cc:304] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:39.372772 9620 leader_election.cc:343] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.372864 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d6db45d2e5c41a3b30cede3301694d3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:39.373106 9620 leader_election.cc:304] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:39.375068 9618 leader_election.cc:343] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.375669 11187 raft_consensus.cc:2749] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:39.376226 11187 raft_consensus.cc:2749] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:39.377837 11274 raft_consensus.cc:515] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:39.380378 9883 tablet.cc:2404] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260504 13:56:39.371152 11234 raft_consensus.cc:515] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.389223 11234 leader_election.cc:290] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:39.390430 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b84c3bedc7a84e0d8a873fb15689b314" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:39.391726 9618 leader_election.cc:343] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:39.393291 9739 tablet.cc:2404] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260504 13:56:39.398705 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dd139c347f7406aac340cc6ac994273" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:39.399237 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6103d953ace4cb8be3c580b3316f87c" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:39.400308 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1a01231bb044433ae5827d07a6592f8" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:39.403708 9932 tablet_bootstrap.cc:700] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6: Time spent opening tablet: real 0.121s user 0.004s sys 0.000s
I20260504 13:56:39.403645 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a89bac060214d30924398007dfa1fa1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:39.404285 9932 tablet_bootstrap.cc:654] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:39.408149 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2bfb681e6824e9f9bdcbea23061d57a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:39.411386 10690 leader_election.cc:336] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781): Timed out: RequestConsensusVote RPC to 127.8.238.252:44781 timed out after 0.664s (SENT)
I20260504 13:56:39.411768 10690 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:39.400359 9931 tablet_bootstrap.cc:654] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:39.421196 10013 tablet_bootstrap.cc:492] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:39.432253 11303 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 13:56:39.432190 11234 raft_consensus.cc:493] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.432718 11234 raft_consensus.cc:515] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:39.434918 11234 leader_election.cc:290] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:39.435902 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c13c4cd2f66426f8e7e81eeac7bc0f3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:39.455351 9618 leader_election.cc:343] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.467248 10013 ts_tablet_manager.cc:1403] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.295s user 0.005s sys 0.005s
I20260504 13:56:39.469949 10013 raft_consensus.cc:359] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:39.482013 11301 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.482508 11301 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:39.485785 11301 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:56:39.486310 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
W20260504 13:56:39.487207 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 114428us
I20260504 13:56:39.487450 10810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
W20260504 13:56:39.487969 9620 consensus_peers.cc:597] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:39.505007 9931 tablet_bootstrap.cc:492] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:39.505625 9931 ts_tablet_manager.cc:1403] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.206s user 0.010s sys 0.001s
I20260504 13:56:39.508002 9931 raft_consensus.cc:359] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.508670 9931 raft_consensus.cc:385] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.508970 9931 raft_consensus.cc:740] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.510076 9931 consensus_queue.cc:260] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.512030 9931 ts_tablet_manager.cc:1434] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260504 13:56:39.521441 9620 leader_election.cc:343] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.521886 9620 leader_election.cc:304] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:39.448127 11274 leader_election.cc:290] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:39.441838 9474 leader_election.cc:343] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:39.522684 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 81005.7us
W20260504 13:56:39.523577 9474 leader_election.cc:343] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.530984 11306 tablet.cc:1724] T 1c12ffa802b840b1937f6f796083c478 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:39.531890 9931 tablet_bootstrap.cc:492] T 89edd71cf5aa4a6abe2084695d3ed807 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:39.537727 9931 tablet_bootstrap.cc:654] T 89edd71cf5aa4a6abe2084695d3ed807 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:39.562774 10013 raft_consensus.cc:385] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.563377 10013 raft_consensus.cc:740] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.564090 10013 consensus_queue.cc:260] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:39.566668 10013 ts_tablet_manager.cc:1434] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.099s user 0.003s sys 0.002s
I20260504 13:56:39.564009 11307 tablet.cc:1724] T 1c12ffa802b840b1937f6f796083c478 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:39.567591 10013 tablet_bootstrap.cc:492] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:39.579524 11234 raft_consensus.cc:2749] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:39.580361 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 58948.9us
W20260504 13:56:39.581774 9620 leader_election.cc:343] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.529672 9932 tablet_bootstrap.cc:492] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:39.582332 9932 ts_tablet_manager.cc:1403] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.335s user 0.008s sys 0.002s
I20260504 13:56:39.584558 9932 raft_consensus.cc:359] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.585186 9932 raft_consensus.cc:385] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.585464 9932 raft_consensus.cc:740] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.586030 9932 consensus_queue.cc:260] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.587947 9932 ts_tablet_manager.cc:1434] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260504 13:56:39.588565 11273 raft_consensus.cc:493] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.589056 11273 raft_consensus.cc:515] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:39.592021 11273 leader_election.cc:290] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:39.488718 9328 leader_election.cc:304] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec9f4c3a43d84fc9a420dd10723c01a1; no voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda
W20260504 13:56:39.593941 9328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 105260us
W20260504 13:56:39.546057 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:39.599459 11301 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260504 13:56:39.610236 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 49697us
W20260504 13:56:39.611404 9474 consensus_peers.cc:597] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:39.612967 9593 tablet.cc:2404] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260504 13:56:39.625888 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c13c4cd2f66426f8e7e81eeac7bc0f3" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:39.645156 9932 tablet_bootstrap.cc:492] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:39.610872 9620 leader_election.cc:304] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:39.649996 11234 raft_consensus.cc:2749] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:39.650718 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 68967.9us
W20260504 13:56:39.651365 9620 leader_election.cc:343] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.651760 9620 leader_election.cc:304] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:39.652395 11234 raft_consensus.cc:2749] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:39.673094 9931 tablet_bootstrap.cc:492] T 89edd71cf5aa4a6abe2084695d3ed807 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:39.673540 9931 ts_tablet_manager.cc:1403] T 89edd71cf5aa4a6abe2084695d3ed807 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.154s user 0.016s sys 0.005s
I20260504 13:56:39.676095 9931 raft_consensus.cc:359] T 89edd71cf5aa4a6abe2084695d3ed807 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.676781 9931 raft_consensus.cc:385] T 89edd71cf5aa4a6abe2084695d3ed807 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.677065 9931 raft_consensus.cc:740] T 89edd71cf5aa4a6abe2084695d3ed807 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.680100 9931 consensus_queue.cc:260] T 89edd71cf5aa4a6abe2084695d3ed807 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.698158 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cb30835150949f39f6e7e7563381763" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:39.698695 9685 raft_consensus.cc:2468] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
W20260504 13:56:39.708582 9474 leader_election.cc:343] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.716403 9931 ts_tablet_manager.cc:1434] T 89edd71cf5aa4a6abe2084695d3ed807 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.043s user 0.006s sys 0.001s
I20260504 13:56:39.717334 9931 tablet_bootstrap.cc:492] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:39.730470 9931 tablet_bootstrap.cc:654] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:39.743462 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6103d953ace4cb8be3c580b3316f87c" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:39.744102 9685 raft_consensus.cc:2468] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:39.746809 9932 tablet_bootstrap.cc:654] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:39.750551 10013 tablet_bootstrap.cc:654] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:39.757634 9931 tablet_bootstrap.cc:492] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:39.758016 9931 ts_tablet_manager.cc:1403] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.005s
I20260504 13:56:39.760478 9931 raft_consensus.cc:359] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.761153 9931 raft_consensus.cc:385] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.761430 9931 raft_consensus.cc:740] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.762089 9931 consensus_queue.cc:260] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.766067 9685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a89bac060214d30924398007dfa1fa1" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:39.766438 9931 ts_tablet_manager.cc:1434] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260504 13:56:39.766629 9685 raft_consensus.cc:3060] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:39.767390 9931 tablet_bootstrap.cc:492] T 7d27b04d3f4443c5ade9616d8626d50e P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:39.803689 9931 tablet_bootstrap.cc:654] T 7d27b04d3f4443c5ade9616d8626d50e P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:39.818979 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c13c4cd2f66426f8e7e81eeac7bc0f3" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:39.819555 9681 raft_consensus.cc:2468] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:39.820261 9931 tablet_bootstrap.cc:492] T 7d27b04d3f4443c5ade9616d8626d50e P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:39.820143 11206 raft_consensus.cc:493] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.820638 11206 raft_consensus.cc:515] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.823081 11206 leader_election.cc:290] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:39.824424 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "982f7c53a1f0482e862b53aeeee7cd25" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:39.824956 9681 raft_consensus.cc:2468] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:39.826020 9768 leader_election.cc:304] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:39.820672 9931 ts_tablet_manager.cc:1403] T 7d27b04d3f4443c5ade9616d8626d50e P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.000s
I20260504 13:56:39.826721 11206 raft_consensus.cc:2804] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:39.827162 11206 raft_consensus.cc:493] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:39.827497 11206 raft_consensus.cc:3060] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:39.829273 9931 raft_consensus.cc:359] T 7d27b04d3f4443c5ade9616d8626d50e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.829792 9931 raft_consensus.cc:385] T 7d27b04d3f4443c5ade9616d8626d50e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.829989 9931 raft_consensus.cc:740] T 7d27b04d3f4443c5ade9616d8626d50e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.830570 9931 consensus_queue.cc:260] T 7d27b04d3f4443c5ade9616d8626d50e P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.833598 11206 raft_consensus.cc:515] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.835999 11234 raft_consensus.cc:493] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.836769 9931 ts_tablet_manager.cc:1434] T 7d27b04d3f4443c5ade9616d8626d50e P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260504 13:56:39.836439 11234 raft_consensus.cc:515] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:39.837559 11206 leader_election.cc:290] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:39.838119 11234 leader_election.cc:290] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:39.838387 9681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "982f7c53a1f0482e862b53aeeee7cd25" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:39.838879 9681 raft_consensus.cc:3060] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:39.841173 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a862a43c5ac443d9bac98e0472ca23b2" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:39.842800 9618 leader_election.cc:343] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.846431 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "982f7c53a1f0482e862b53aeeee7cd25" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:39.849310 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "982f7c53a1f0482e862b53aeeee7cd25" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:39.849879 11206 raft_consensus.cc:493] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.850268 11206 raft_consensus.cc:515] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:39.851850 9767 leader_election.cc:343] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.852324 11206 leader_election.cc:290] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:39.854609 9681 raft_consensus.cc:2468] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:39.855626 9768 leader_election.cc:304] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:39.856402 11255 raft_consensus.cc:2804] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:39.856745 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59f9115b286d44f0b7eac7c74275cb21" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:39.856861 11255 raft_consensus.cc:697] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:39.857278 9684 raft_consensus.cc:2468] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:39.857669 11255 consensus_queue.cc:237] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.859733 9768 leader_election.cc:304] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:39.860320 11206 raft_consensus.cc:2804] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:39.860606 11206 raft_consensus.cc:493] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:39.860898 11206 raft_consensus.cc:3060] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:39.863004 10013 tablet_bootstrap.cc:492] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:39.863428 10013 ts_tablet_manager.cc:1403] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.296s user 0.007s sys 0.003s
I20260504 13:56:39.865898 9931 tablet_bootstrap.cc:492] T 538c8c2412104c3e8b19394a1cc38818 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:39.865698 10013 raft_consensus.cc:359] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:39.866323 10013 raft_consensus.cc:385] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.866721 10013 raft_consensus.cc:740] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.869053 11206 raft_consensus.cc:515] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.875531 11206 leader_election.cc:290] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:39.876370 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d6db45d2e5c41a3b30cede3301694d3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:39.877331 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c13c4cd2f66426f8e7e81eeac7bc0f3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:39.879045 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b84c3bedc7a84e0d8a873fb15689b314" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:39.880498 9767 leader_election.cc:343] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.884436 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59f9115b286d44f0b7eac7c74275cb21" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:39.884922 9683 raft_consensus.cc:3060] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:39.774024 9685 raft_consensus.cc:2468] T 7a89bac060214d30924398007dfa1fa1 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:39.892402 9683 raft_consensus.cc:2468] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:39.893386 9768 leader_election.cc:304] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:39.894059 11206 raft_consensus.cc:2804] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:39.894515 11206 raft_consensus.cc:697] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:39.895267 11206 consensus_queue.cc:237] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.901772 9932 tablet_bootstrap.cc:492] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:39.902216 9932 ts_tablet_manager.cc:1403] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.258s user 0.008s sys 0.000s
I20260504 13:56:39.905575 9932 raft_consensus.cc:359] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.906136 9932 raft_consensus.cc:385] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.906440 9932 raft_consensus.cc:740] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.911615 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59f9115b286d44f0b7eac7c74275cb21" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:39.913484 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59f9115b286d44f0b7eac7c74275cb21" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:39.915740 9931 tablet_bootstrap.cc:654] T 538c8c2412104c3e8b19394a1cc38818 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:39.927847 9186 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
I20260504 13:56:39.928823 9186 catalog_manager.cc:5671] T 89edd71cf5aa4a6abe2084695d3ed807 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:39.929714 11234 raft_consensus.cc:493] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.930209 11234 raft_consensus.cc:515] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.930843 9186 catalog_manager.cc:5671] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:39.931836 9186 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38]). Expected version 1 got 0
I20260504 13:56:39.932312 9931 tablet_bootstrap.cc:492] T 538c8c2412104c3e8b19394a1cc38818 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:39.932403 11234 leader_election.cc:290] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:39.932756 9931 ts_tablet_manager.cc:1403] T 538c8c2412104c3e8b19394a1cc38818 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.001s
I20260504 13:56:39.932549 9186 catalog_manager.cc:5671] T 2a629c2a69084b6b98984c8d374eeb1e P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:39.934383 9186 catalog_manager.cc:5671] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:39.936924 9931 raft_consensus.cc:359] T 538c8c2412104c3e8b19394a1cc38818 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.937613 9931 raft_consensus.cc:385] T 538c8c2412104c3e8b19394a1cc38818 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.937885 9931 raft_consensus.cc:740] T 538c8c2412104c3e8b19394a1cc38818 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.942026 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d27b04d3f4443c5ade9616d8626d50e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:39.942106 9931 consensus_queue.cc:260] T 538c8c2412104c3e8b19394a1cc38818 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:39.943497 9618 leader_election.cc:343] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.944118 9931 ts_tablet_manager.cc:1434] T 538c8c2412104c3e8b19394a1cc38818 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
I20260504 13:56:39.945082 9931 tablet_bootstrap.cc:492] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:39.949818 9473 leader_election.cc:304] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:39.950062 11234 raft_consensus.cc:493] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.950623 11234 raft_consensus.cc:515] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.952651 11234 leader_election.cc:290] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:39.953475 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7989338bc4bf4bad8fb8fe37587d0a16" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:39.954767 9618 leader_election.cc:343] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:39.952479 11274 raft_consensus.cc:2804] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:39.955490 11274 raft_consensus.cc:493] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:39.955893 11274 raft_consensus.cc:3060] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:39.967523 9931 tablet_bootstrap.cc:654] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:39.975898 9473 leader_election.cc:304] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:39.977258 10956 raft_consensus.cc:2804] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:39.977558 10956 raft_consensus.cc:493] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:39.977821 10956 raft_consensus.cc:3060] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:39.979450 11206 raft_consensus.cc:493] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:39.979864 11206 raft_consensus.cc:515] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.980387 9931 tablet_bootstrap.cc:492] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:39.980811 9931 ts_tablet_manager.cc:1403] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.000s
I20260504 13:56:39.981566 11206 leader_election.cc:290] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:39.983162 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d9bd2722d23420ea0eac7664357a60c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:39.983335 9931 raft_consensus.cc:359] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.983744 9683 raft_consensus.cc:2468] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:39.983979 9931 raft_consensus.cc:385] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:39.984236 9931 raft_consensus.cc:740] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:39.984750 9768 leader_election.cc:304] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:39.935865 9186 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet dffddd4b89984c54bc26dcb4a9488525 (table default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568]). Expected version 1 got 0
I20260504 13:56:39.985409 11206 raft_consensus.cc:2804] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:39.985720 11206 raft_consensus.cc:493] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:39.986018 11206 raft_consensus.cc:3060] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:39.986023 9186 catalog_manager.cc:5671] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:39.993580 9931 consensus_queue.cc:260] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.993966 11206 raft_consensus.cc:515] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.995998 9931 ts_tablet_manager.cc:1434] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260504 13:56:39.996851 9931 tablet_bootstrap.cc:492] T cc2f95dadd544f57bd6d1467d30e954d P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:39.996970 10956 raft_consensus.cc:515] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:39.998569 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d9bd2722d23420ea0eac7664357a60c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:39.999082 9683 raft_consensus.cc:3060] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:39.999840 10956 leader_election.cc:290] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:40.004698 9767 leader_election.cc:343] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:40.005297 11206 leader_election.cc:290] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:40.005586 9767 leader_election.cc:343] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:40.015024 10013 consensus_queue.cc:260] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:40.015518 9683 raft_consensus.cc:2468] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:40.020260 11274 raft_consensus.cc:515] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.021188 9768 leader_election.cc:304] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:40.026032 11206 raft_consensus.cc:2804] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:40.027086 11206 raft_consensus.cc:697] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:40.028107 11206 consensus_queue.cc:237] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.031590 11325 consensus_queue.cc:1048] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:39.987048 9186 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet de6714ee86d64112baf13dd1b2b87582 (table default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469]). Expected version 1 got 0
I20260504 13:56:40.036367 10013 ts_tablet_manager.cc:1434] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.173s user 0.005s sys 0.000s
I20260504 13:56:40.038136 10013 tablet_bootstrap.cc:492] T e53381ee0e7f4e9781b833b435bca831 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:40.043077 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a862a43c5ac443d9bac98e0472ca23b2" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:40.051481 10013 tablet_bootstrap.cc:654] T e53381ee0e7f4e9781b833b435bca831 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:40.051709 9473 leader_election.cc:304] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:40.052376 11325 raft_consensus.cc:2804] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:40.052760 11325 raft_consensus.cc:493] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:40.053061 11325 raft_consensus.cc:3060] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:40.055186 11206 raft_consensus.cc:493] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:40.055763 11206 raft_consensus.cc:515] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:40.060209 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88619eaa917d4c33a910fc1347ab126e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:40.060838 9683 raft_consensus.cc:2468] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:40.061934 9768 leader_election.cc:304] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:40.063081 11253 raft_consensus.cc:2804] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:40.063509 11253 raft_consensus.cc:493] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:40.063898 11253 raft_consensus.cc:3060] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:40.067582 9931 tablet_bootstrap.cc:654] T cc2f95dadd544f57bd6d1467d30e954d P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:40.072194 11253 raft_consensus.cc:515] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:40.075947 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88619eaa917d4c33a910fc1347ab126e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:40.076442 9683 raft_consensus.cc:3060] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:40.083276 11206 leader_election.cc:290] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:40.084648 9683 raft_consensus.cc:2468] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:40.085790 9768 leader_election.cc:304] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:40.089452 11206 raft_consensus.cc:2804] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:40.089918 11206 raft_consensus.cc:697] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:40.090828 11206 consensus_queue.cc:237] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:40.099565 11325 raft_consensus.cc:515] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:40.101341 9932 consensus_queue.cc:260] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.102803 9932 ts_tablet_manager.cc:1434] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.200s user 0.002s sys 0.005s
I20260504 13:56:40.107265 11253 leader_election.cc:290] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:40.119874 9932 tablet_bootstrap.cc:492] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:40.100795 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 84649.5us
I20260504 13:56:40.133147 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6103d953ace4cb8be3c580b3316f87c" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:40.134698 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cb30835150949f39f6e7e7563381763" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:40.134914 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 13:56:40.108027 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 56389.2us
I20260504 13:56:40.138976 9931 tablet_bootstrap.cc:492] T cc2f95dadd544f57bd6d1467d30e954d P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:40.139604 9931 ts_tablet_manager.cc:1403] T cc2f95dadd544f57bd6d1467d30e954d P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.143s user 0.010s sys 0.000s
I20260504 13:56:40.140982 9828 raft_consensus.cc:3060] T dffddd4b89984c54bc26dcb4a9488525 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:40.141917 9473 leader_election.cc:304] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:40.142167 9931 raft_consensus.cc:359] T cc2f95dadd544f57bd6d1467d30e954d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.142876 9931 raft_consensus.cc:385] T cc2f95dadd544f57bd6d1467d30e954d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:40.143018 11326 raft_consensus.cc:2804] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:40.143106 9931 raft_consensus.cc:740] T cc2f95dadd544f57bd6d1467d30e954d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:40.143792 9931 consensus_queue.cc:260] T cc2f95dadd544f57bd6d1467d30e954d P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.148407 11326 raft_consensus.cc:697] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:40.151497 11273 raft_consensus.cc:493] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:40.151582 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d27b04d3f4443c5ade9616d8626d50e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:40.152192 9832 raft_consensus.cc:2468] T 7d27b04d3f4443c5ade9616d8626d50e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:40.151949 11273 raft_consensus.cc:515] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:40.152397 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d9bd2722d23420ea0eac7664357a60c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:40.153210 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7989338bc4bf4bad8fb8fe37587d0a16" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:40.153688 11273 leader_election.cc:290] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:40.154004 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d9bd2722d23420ea0eac7664357a60c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:40.156622 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88619eaa917d4c33a910fc1347ab126e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:40.158041 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88619eaa917d4c33a910fc1347ab126e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:40.149170 11326 consensus_queue.cc:237] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:40.159278 9767 leader_election.cc:343] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:40.163020 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a862a43c5ac443d9bac98e0472ca23b2" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:40.165129 9767 leader_election.cc:343] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:40.166520 9767 leader_election.cc:343] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:40.168061 9767 leader_election.cc:343] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:40.171805 9474 leader_election.cc:343] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:40.172533 9474 leader_election.cc:343] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:40.182403 9932 tablet_bootstrap.cc:654] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:40.183800 9661 tablet_service.cc:1558] Processing DeleteTablet for tablet 21d6358b451d4b8d87a22e30e0780f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:25 UTC) from {username='slave'} at 127.0.0.1:34768
I20260504 13:56:40.185086 11325 leader_election.cc:290] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:40.188465 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cb30835150949f39f6e7e7563381763" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:40.188957 9684 raft_consensus.cc:3060] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:40.189504 9474 leader_election.cc:343] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:40.197001 9328 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c47b6e3b4d0c45c28384f7aae6b749de consensus state has no leader
I20260504 13:56:40.197963 9931 ts_tablet_manager.cc:1434] T cc2f95dadd544f57bd6d1467d30e954d P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.058s user 0.005s sys 0.000s
I20260504 13:56:40.127413 11274 leader_election.cc:290] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:40.203991 9931 tablet_bootstrap.cc:492] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:40.201810 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c13c4cd2f66426f8e7e81eeac7bc0f3" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:40.211922 9474 leader_election.cc:343] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:40.231634 11206 raft_consensus.cc:493] T 89edd71cf5aa4a6abe2084695d3ed807 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:40.232059 11206 raft_consensus.cc:515] T 89edd71cf5aa4a6abe2084695d3ed807 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.233798 11206 leader_election.cc:290] T 89edd71cf5aa4a6abe2084695d3ed807 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:40.235601 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89edd71cf5aa4a6abe2084695d3ed807" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:40.242265 9508 tablet_service.cc:1511] Processing CreateTablet for tablet e9e66dacf5794b6195575556be889bce (DEFAULT_TABLE table=default.Table_d6a5a22d78264e799e80d31d5749600a [id=d63acf266ca440729128a717eed32433]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:40.243397 9508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e66dacf5794b6195575556be889bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:40.247529 9931 tablet_bootstrap.cc:654] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:40.273216 11234 raft_consensus.cc:493] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:40.273669 11234 raft_consensus.cc:515] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.275424 11234 leader_election.cc:290] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:40.276810 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc2f95dadd544f57bd6d1467d30e954d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:40.279405 9186 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:45997 (request call id 267) took 1125 ms (client timeout 1000 ms). Trace:
W20260504 13:56:40.283779 9618 leader_election.cc:343] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:40.287159 9175 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 57983fcecdb64a42884f797b4db57df0 consensus state has no leader
I20260504 13:56:40.292541 9677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6103d953ace4cb8be3c580b3316f87c" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:40.293071 9677 raft_consensus.cc:3060] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:40.293572 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c13c4cd2f66426f8e7e81eeac7bc0f3" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:40.294099 9682 raft_consensus.cc:3060] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:40.299803 9517 tablet_service.cc:1511] Processing CreateTablet for tablet b7b942037c0e45a6b3cccba288b90b00 (DEFAULT_TABLE table=default.Table_dd11ea89ad7248c39ec114af58cf598f [id=14bd1983ba95468e9e70d8f5724f42e6]), partition=RANGE (key) PARTITION UNBOUNDED
W20260504 13:56:40.301507 9620 leader_election.cc:343] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:40.301941 9620 leader_election.cc:304] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:40.304281 9517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b942037c0e45a6b3cccba288b90b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:40.315598 9507 tablet_service.cc:1511] Processing CreateTablet for tablet 564d0fafb49a4b4bb3bdb240c74c3410 (DEFAULT_TABLE table=default.Table_1c9b2e63142440a5a45018d187a6cded [id=8a427e40b3374dafa140c9156edcc660]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:40.316898 9507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 564d0fafb49a4b4bb3bdb240c74c3410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:40.322611 10013 tablet_bootstrap.cc:492] T e53381ee0e7f4e9781b833b435bca831 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:40.323060 10013 ts_tablet_manager.cc:1403] T e53381ee0e7f4e9781b833b435bca831 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.285s user 0.017s sys 0.012s
I20260504 13:56:40.325688 10013 raft_consensus.cc:359] T e53381ee0e7f4e9781b833b435bca831 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.326285 10013 raft_consensus.cc:385] T e53381ee0e7f4e9781b833b435bca831 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:40.328084 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a862a43c5ac443d9bac98e0472ca23b2" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:40.328650 9683 raft_consensus.cc:2468] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:40.329380 11234 raft_consensus.cc:2749] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:40.331769 9671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89edd71cf5aa4a6abe2084695d3ed807" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:40.333884 11338 tablet_replica.cc:333] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:40.334957 11338 raft_consensus.cc:2243] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:40.335472 11338 raft_consensus.cc:2272] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:40.344653 9684 raft_consensus.cc:2468] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:40.356295 9832 raft_consensus.cc:3060] T 538c8c2412104c3e8b19394a1cc38818 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:40.353133 9677 raft_consensus.cc:2468] T b6103d953ace4cb8be3c580b3316f87c P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:40.365543 9682 raft_consensus.cc:2468] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:40.361967 11253 raft_consensus.cc:493] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:40.370172 11253 raft_consensus.cc:515] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.371883 11253 leader_election.cc:290] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:40.403136 11333 raft_consensus.cc:493] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:40.403689 11333 raft_consensus.cc:515] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.406877 11333 leader_election.cc:290] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:40.417603 11346 tablet.cc:1724] T 1c12ffa802b840b1937f6f796083c478 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260504 13:56:40.424978 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 123507us
W20260504 13:56:40.426043 9620 leader_election.cc:343] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:40.279699 9186 rpcz_store.cc:269] 0504 13:56:39.153579 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:39.189115 (+ 35536us) service_pool.cc:224] Handling call
0504 13:56:40.279343 (+1090228us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:40.037707 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:40.038139 (+ 432us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:40.038166 (+ 27us) write_op.cc:435] Acquired schema lock
0504 13:56:40.038182 (+ 16us) tablet.cc:662] Decoding operations
0504 13:56:40.038741 (+ 559us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:40.038782 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:40.038807 (+ 25us) tablet.cc:685] Acquiring locks for 5 operations
0504 13:56:40.039261 (+ 454us) tablet.cc:701] Row locks acquired
0504 13:56:40.039278 (+ 17us) write_op.cc:260] PREPARE: finished
0504 13:56:40.039396 (+ 118us) write_op.cc:270] Start()
0504 13:56:40.039475 (+ 79us) write_op.cc:276] Timestamp: P: 1777903000039203 usec, L: 0
0504 13:56:40.039495 (+ 20us) op_driver.cc:348] REPLICATION: starting
0504 13:56:40.040044 (+ 549us) log.cc:844] Serialized 2054 byte log entry
0504 13:56:40.048901 (+ 8857us) op_driver.cc:464] REPLICATION: finished
0504 13:56:40.049184 (+ 283us) write_op.cc:301] APPLY: starting
0504 13:56:40.049234 (+ 50us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:40.049373 (+ 139us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:40.049454 (+ 81us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:40.049898 (+ 444us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:40.050049 (+ 151us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0504 13:56:40.050090 (+ 41us) write_op.cc:312] APPLY: finished
0504 13:56:40.050280 (+ 190us) log.cc:844] Serialized 47 byte log entry
0504 13:56:40.050940 (+ 660us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:40.051026 (+ 86us) write_op.cc:454] Released schema lock
0504 13:56:40.051056 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:40.051283 (+ 227us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:40.051305 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":100,"apply.run_cpu_time_us":1599,"apply.run_wall_time_us":2092,"num_ops":5,"prepare.queue_time_us":102,"prepare.run_cpu_time_us":2611,"prepare.run_wall_time_us":3790,"raft.queue_time_us":171,"raft.run_cpu_time_us":648,"raft.run_wall_time_us":647,"replication_time_us":9316}]]}
I20260504 13:56:40.375591 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ffeae4e5af4cc8a3eb3bc0bedf18cc" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:40.441768 10013 raft_consensus.cc:740] T e53381ee0e7f4e9781b833b435bca831 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:40.442443 10013 consensus_queue.cc:260] T e53381ee0e7f4e9781b833b435bca831 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.444694 10013 ts_tablet_manager.cc:1434] T e53381ee0e7f4e9781b833b435bca831 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.121s user 0.002s sys 0.004s
I20260504 13:56:40.401897 11352 tablet.cc:1724] T dffddd4b89984c54bc26dcb4a9488525 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:40.406463 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf11f17e95dc411294c755e17c57811e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:40.430567 9505 tablet_service.cc:1511] Processing CreateTablet for tablet f337e9698fe64282a684e90d9c61da5a (DEFAULT_TABLE table=default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:40.447691 9620 leader_election.cc:304] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:40.447907 9507 tablet_service.cc:1511] Processing CreateTablet for tablet 23ff0d52a90c4056880190dbd936cb8e (DEFAULT_TABLE table=default.Table_24398dd8ee63413ebb585587353ea155 [id=1d2bbdaf6382433cb8c2a55f62f0388d]), partition=RANGE (key) PARTITION UNBOUNDED
W20260504 13:56:40.448900 9620 leader_election.cc:343] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:40.449276 9620 leader_election.cc:304] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:40.449959 11234 raft_consensus.cc:2749] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:40.450457 11234 raft_consensus.cc:2749] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:40.373363 9932 tablet_bootstrap.cc:492] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:40.430648 11333 raft_consensus.cc:493] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:40.451306 11333 raft_consensus.cc:515] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:40.452940 11333 leader_election.cc:290] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:40.371676 11338 ts_tablet_manager.cc:1916] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260504 13:56:40.446959 9618 leader_election.cc:343] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:40.461958 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af5c5a6cae4e4b6ea3eece2e21a587ec" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:40.463315 9618 leader_election.cc:343] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:40.463297 11333 raft_consensus.cc:493] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:40.464257 11333 raft_consensus.cc:515] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.464615 10013 tablet_bootstrap.cc:492] T 3d9bd2722d23420ea0eac7664357a60c P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:40.464895 9507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23ff0d52a90c4056880190dbd936cb8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:40.465742 9505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f337e9698fe64282a684e90d9c61da5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:40.467041 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e53381ee0e7f4e9781b833b435bca831" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:40.467649 9535 raft_consensus.cc:2468] T e53381ee0e7f4e9781b833b435bca831 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:40.469029 9618 leader_election.cc:304] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:40.469694 11333 leader_election.cc:290] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:40.470198 11333 raft_consensus.cc:2804] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:40.470534 11333 raft_consensus.cc:493] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:40.470860 11333 raft_consensus.cc:3060] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:40.478863 10013 tablet_bootstrap.cc:654] T 3d9bd2722d23420ea0eac7664357a60c P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:40.479641 9932 ts_tablet_manager.cc:1403] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.360s user 0.007s sys 0.012s
W20260504 13:56:40.455101 9767 leader_election.cc:343] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:40.509644 9797 tablet_service.cc:1511] Processing CreateTablet for tablet 564d0fafb49a4b4bb3bdb240c74c3410 (DEFAULT_TABLE table=default.Table_1c9b2e63142440a5a45018d187a6cded [id=8a427e40b3374dafa140c9156edcc660]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:40.510090 9805 tablet_service.cc:1511] Processing CreateTablet for tablet 23ff0d52a90c4056880190dbd936cb8e (DEFAULT_TABLE table=default.Table_24398dd8ee63413ebb585587353ea155 [id=1d2bbdaf6382433cb8c2a55f62f0388d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:40.511042 9801 tablet_service.cc:1511] Processing CreateTablet for tablet b7b942037c0e45a6b3cccba288b90b00 (DEFAULT_TABLE table=default.Table_dd11ea89ad7248c39ec114af58cf598f [id=14bd1983ba95468e9e70d8f5724f42e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:40.512566 9806 tablet_service.cc:1511] Processing CreateTablet for tablet f337e9698fe64282a684e90d9c61da5a (DEFAULT_TABLE table=default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:40.514479 9795 tablet_service.cc:1511] Processing CreateTablet for tablet e9e66dacf5794b6195575556be889bce (DEFAULT_TABLE table=default.Table_d6a5a22d78264e799e80d31d5749600a [id=d63acf266ca440729128a717eed32433]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:40.511054 9797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 564d0fafb49a4b4bb3bdb240c74c3410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:40.517015 9881 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:40.527985 11354 tablet.cc:1724] T dffddd4b89984c54bc26dcb4a9488525 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:40.529397 9805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23ff0d52a90c4056880190dbd936cb8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:40.482628 9932 raft_consensus.cc:359] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:40.532966 9932 raft_consensus.cc:385] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:40.533208 9932 raft_consensus.cc:740] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:40.538342 9806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f337e9698fe64282a684e90d9c61da5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:40.542508 9795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e66dacf5794b6195575556be889bce. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:40.544023 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 93777.5us
I20260504 13:56:40.544700 9801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b942037c0e45a6b3cccba288b90b00. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:56:40.545527 9620 leader_election.cc:343] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:40.545876 9620 leader_election.cc:304] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:40.549556 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:40.533735 9932 consensus_queue.cc:260] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:40.560042 11347 raft_consensus.cc:2749] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:40.562541 9932 ts_tablet_manager.cc:1434] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.083s user 0.004s sys 0.002s
I20260504 13:56:40.566542 9620 leader_election.cc:304] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:40.569140 11333 raft_consensus.cc:515] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.573005 11347 raft_consensus.cc:2804] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:40.573348 11347 raft_consensus.cc:493] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:40.573671 11347 raft_consensus.cc:3060] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:40.575266 10013 tablet_bootstrap.cc:492] T 3d9bd2722d23420ea0eac7664357a60c P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:40.575685 10013 ts_tablet_manager.cc:1403] T 3d9bd2722d23420ea0eac7664357a60c P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.111s user 0.011s sys 0.020s
I20260504 13:56:40.577040 9931 tablet_bootstrap.cc:492] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:40.577452 9931 ts_tablet_manager.cc:1403] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.374s user 0.008s sys 0.009s
I20260504 13:56:40.580543 11347 raft_consensus.cc:515] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.581492 10013 raft_consensus.cc:359] T 3d9bd2722d23420ea0eac7664357a60c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:40.582989 9620 leader_election.cc:343] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:40.583405 9620 leader_election.cc:304] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:40.584334 11355 raft_consensus.cc:2749] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:40.581573 11352 tablet.cc:1724] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:40.587986 10013 raft_consensus.cc:385] T 3d9bd2722d23420ea0eac7664357a60c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:40.595329 10013 raft_consensus.cc:740] T 3d9bd2722d23420ea0eac7664357a60c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:40.595953 10013 consensus_queue.cc:260] T 3d9bd2722d23420ea0eac7664357a60c P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.597816 10013 ts_tablet_manager.cc:1434] T 3d9bd2722d23420ea0eac7664357a60c P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20260504 13:56:40.620335 11352 tablet.cc:1724] T de6714ee86d64112baf13dd1b2b87582 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:40.622046 10013 tablet_bootstrap.cc:492] T 59f9115b286d44f0b7eac7c74275cb21 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:40.634543 11354 tablet.cc:1724] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:40.638005 9932 tablet_bootstrap.cc:492] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:40.676524 11333 leader_election.cc:290] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:40.677816 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e53381ee0e7f4e9781b833b435bca831" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:40.678335 9530 raft_consensus.cc:3060] T e53381ee0e7f4e9781b833b435bca831 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:40.684732 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc2f95dadd544f57bd6d1467d30e954d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:40.684968 9530 raft_consensus.cc:2468] T e53381ee0e7f4e9781b833b435bca831 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:40.685524 9832 raft_consensus.cc:2468] T cc2f95dadd544f57bd6d1467d30e954d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:40.665540 11360 tablet.cc:1724] T de6714ee86d64112baf13dd1b2b87582 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:40.687407 9618 leader_election.cc:304] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:40.690685 11355 raft_consensus.cc:2804] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:40.691604 11355 raft_consensus.cc:697] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:40.692396 11355 consensus_queue.cc:237] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.708415 11333 raft_consensus.cc:493] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:40.709093 11333 raft_consensus.cc:515] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:40.678730 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 317160us
I20260504 13:56:40.712780 9768 leader_election.cc:304] T 89edd71cf5aa4a6abe2084695d3ed807 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:40.713413 11255 raft_consensus.cc:2749] T 89edd71cf5aa4a6abe2084695d3ed807 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:40.715317 9473 leader_election.cc:304] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:40.716424 11324 raft_consensus.cc:2804] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:40.716755 11324 raft_consensus.cc:493] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:40.717051 11324 raft_consensus.cc:3060] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:40.723325 11347 leader_election.cc:290] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:40.738157 10013 tablet_bootstrap.cc:654] T 59f9115b286d44f0b7eac7c74275cb21 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:40.746479 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ffeae4e5af4cc8a3eb3bc0bedf18cc" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:40.747740 9618 leader_election.cc:343] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:40.752267 11333 leader_election.cc:290] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:40.753806 9473 leader_election.cc:304] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:40.754591 11325 raft_consensus.cc:2804] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:40.756989 11325 raft_consensus.cc:697] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:40.757956 9473 leader_election.cc:304] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:40.757819 11325 consensus_queue.cc:237] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.758706 11274 raft_consensus.cc:2804] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:40.759116 11274 raft_consensus.cc:697] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:40.759902 11274 consensus_queue.cc:237] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.761983 9588 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:40.769057 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d27b04d3f4443c5ade9616d8626d50e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:40.783799 9931 raft_consensus.cc:359] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:40.784657 9931 raft_consensus.cc:385] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:40.785006 9931 raft_consensus.cc:740] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:40.785826 9931 consensus_queue.cc:260] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:40.787897 11368 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:40.788300 11368 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:40.790848 11368 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:56:40.791469 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:40.792841 10810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:40.796094 11324 raft_consensus.cc:515] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:40.711064 9187 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 88619eaa917d4c33a910fc1347ab126e (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:40.808476 9187 catalog_manager.cc:5671] T 4fae9274bb8240798f85b79504a227b3 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:40.753067 9932 tablet_bootstrap.cc:654] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:40.755028 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ffeae4e5af4cc8a3eb3bc0bedf18cc" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:40.811964 9931 ts_tablet_manager.cc:1434] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.234s user 0.007s sys 0.001s
W20260504 13:56:40.813643 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 62071.6us
I20260504 13:56:40.818331 11255 consensus_queue.cc:1048] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:40.797145 9328 leader_election.cc:304] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec9f4c3a43d84fc9a420dd10723c01a1; no voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda
I20260504 13:56:40.833011 11368 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 13:56:40.848486 9187 catalog_manager.cc:5671] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:40.850368 9187 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4]). Expected version 1 got 0
I20260504 13:56:40.852020 9187 catalog_manager.cc:5671] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:40.863829 9931 tablet_bootstrap.cc:492] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:40.870715 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 52528.1us
W20260504 13:56:40.877338 9328 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 80243us
W20260504 13:56:40.883623 9618 leader_election.cc:343] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:40.884859 11255 consensus_queue.cc:1048] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:40.885383 9187 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
I20260504 13:56:40.890224 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf11f17e95dc411294c755e17c57811e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:40.890793 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af5c5a6cae4e4b6ea3eece2e21a587ec" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:40.891526 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e53381ee0e7f4e9781b833b435bca831" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:40.895803 9187 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38]). Expected version 1 got 0
I20260504 13:56:40.896988 9661 tablet_service.cc:1511] Processing CreateTablet for tablet e9e66dacf5794b6195575556be889bce (DEFAULT_TABLE table=default.Table_d6a5a22d78264e799e80d31d5749600a [id=d63acf266ca440729128a717eed32433]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:40.905449 9661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e66dacf5794b6195575556be889bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:40.886644 9682 raft_consensus.cc:2468] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:40.906468 9187 catalog_manager.cc:5671] T bae90898739b4303af0608e39e8c0367 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:40.908135 9187 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 3 got 0
I20260504 13:56:40.908766 9187 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 1c12ffa802b840b1937f6f796083c478 (table default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1]). Expected version 1 got 0
I20260504 13:56:40.909008 9768 leader_election.cc:304] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:40.909431 9187 catalog_manager.cc:5671] T 4cf1504f0202481ba8b08f9bec5bc3d9 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:40.912192 9660 tablet_service.cc:1511] Processing CreateTablet for tablet b7b942037c0e45a6b3cccba288b90b00 (DEFAULT_TABLE table=default.Table_dd11ea89ad7248c39ec114af58cf598f [id=14bd1983ba95468e9e70d8f5724f42e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:40.913102 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b942037c0e45a6b3cccba288b90b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:40.914251 11338 ts_tablet_manager.cc:1929] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:40.914610 11338 log.cc:1199] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/21d6358b451d4b8d87a22e30e0780f44
I20260504 13:56:40.915609 11338 ts_tablet_manager.cc:1950] T 21d6358b451d4b8d87a22e30e0780f44 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:40.915792 9659 tablet_service.cc:1511] Processing CreateTablet for tablet f337e9698fe64282a684e90d9c61da5a (DEFAULT_TABLE table=default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:40.916920 9666 tablet_service.cc:1511] Processing CreateTablet for tablet 564d0fafb49a4b4bb3bdb240c74c3410 (DEFAULT_TABLE table=default.Table_1c9b2e63142440a5a45018d187a6cded [id=8a427e40b3374dafa140c9156edcc660]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:40.918874 9657 tablet_service.cc:1511] Processing CreateTablet for tablet 23ff0d52a90c4056880190dbd936cb8e (DEFAULT_TABLE table=default.Table_24398dd8ee63413ebb585587353ea155 [id=1d2bbdaf6382433cb8c2a55f62f0388d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:40.933806 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e53381ee0e7f4e9781b833b435bca831" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:40.937566 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d27b04d3f4443c5ade9616d8626d50e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:40.938141 9833 raft_consensus.cc:3060] T 7d27b04d3f4443c5ade9616d8626d50e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:40.938252 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ffeae4e5af4cc8a3eb3bc0bedf18cc" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:40.938908 9829 raft_consensus.cc:2468] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:40.939769 11206 raft_consensus.cc:2804] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:40.940141 11206 raft_consensus.cc:493] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:40.940538 11206 raft_consensus.cc:3060] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:40.945752 9833 raft_consensus.cc:2468] T 7d27b04d3f4443c5ade9616d8626d50e P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:40.948817 11206 raft_consensus.cc:515] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.955946 9659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f337e9698fe64282a684e90d9c61da5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:40.955150 9174 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet 21d6358b451d4b8d87a22e30e0780f44 (table default.Table_5bfac12ab2f84c06b1f992848d8f5041 [id=c2f8dc1bfbc14f8a8c093490ff05a18c]) successfully deleted
I20260504 13:56:40.911134 9187 catalog_manager.cc:5671] T 2f6b3e969760442283c3191e062c1839 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:40.964537 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 564d0fafb49a4b4bb3bdb240c74c3410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:40.964169 9187 catalog_manager.cc:5671] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:40.965817 9187 catalog_manager.cc:5671] T a26c6a2ffb8746a3941a56c7259412e3 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:40.979476 9657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23ff0d52a90c4056880190dbd936cb8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:40.951238 11349 raft_consensus.cc:493] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:40.956851 11253 raft_consensus.cc:493] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:40.980003 11253 raft_consensus.cc:515] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:40.981854 11253 leader_election.cc:290] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:40.975730 9187 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet de6714ee86d64112baf13dd1b2b87582 (table default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469]). Expected version 1 got 0
I20260504 13:56:40.982525 11349 raft_consensus.cc:515] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:40.983155 9187 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 2ddf890c17964e7f90409ebe3c4034ef (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]). Expected version 2 got 0
I20260504 13:56:40.983959 9187 catalog_manager.cc:5671] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:40.984975 11349 leader_election.cc:290] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:40.985472 9187 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]). Expected version 1 got 0
I20260504 13:56:40.987017 11206 leader_election.cc:290] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:40.987097 9187 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet e53381ee0e7f4e9781b833b435bca831 (table default.Table_9b40ba316993413181a9743fb7bab7fb [id=fb4e5ce5526e44de8effdd766f4c257f]). Expected version 1 got 0
I20260504 13:56:40.891430 9473 leader_election.cc:304] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:41.002233 9932 tablet_bootstrap.cc:492] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:41.002715 9932 ts_tablet_manager.cc:1403] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.368s user 0.013s sys 0.004s
I20260504 13:56:40.975493 11234 raft_consensus.cc:493] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:40.985527 11355 raft_consensus.cc:493] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:41.010476 11369 raft_consensus.cc:2804] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:41.021221 11369 raft_consensus.cc:697] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:41.022133 11369 consensus_queue.cc:237] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:41.024648 11324 leader_election.cc:290] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:41.040571 9187 catalog_manager.cc:5671] T f3e86027e3e74bc196e9a9ffc9fdf8a5 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:41.044773 9187 catalog_manager.cc:5671] T fac3e36cfa4848a1ab015f89c56c2740 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:41.046072 9187 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet dffddd4b89984c54bc26dcb4a9488525 (table default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568]). Expected version 1 got 0
I20260504 13:56:41.046927 9187 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]). Expected version 1 got 0
I20260504 13:56:41.063709 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e819494171485e9af8bc8ec33a6ff0" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:41.064484 9684 raft_consensus.cc:2468] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
W20260504 13:56:41.080688 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 189310us
I20260504 13:56:41.018344 11234 raft_consensus.cc:515] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.097608 9768 leader_election.cc:304] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:41.101245 11381 raft_consensus.cc:2804] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:41.101596 11381 raft_consensus.cc:493] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:41.101903 11381 raft_consensus.cc:3060] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:41.112788 10013 tablet_bootstrap.cc:492] T 59f9115b286d44f0b7eac7c74275cb21 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:41.113171 10013 ts_tablet_manager.cc:1403] T 59f9115b286d44f0b7eac7c74275cb21 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.491s user 0.037s sys 0.065s
I20260504 13:56:41.026881 11355 raft_consensus.cc:515] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:41.123700 9932 raft_consensus.cc:359] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.126739 9932 raft_consensus.cc:385] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:41.127045 9932 raft_consensus.cc:740] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:41.127749 9932 consensus_queue.cc:260] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.131016 9931 tablet_bootstrap.cc:654] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:41.142591 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:41.165092 11347 raft_consensus.cc:493] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:41.165531 11347 raft_consensus.cc:515] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:41.154855 10013 raft_consensus.cc:359] T 59f9115b286d44f0b7eac7c74275cb21 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.166915 10013 raft_consensus.cc:385] T 59f9115b286d44f0b7eac7c74275cb21 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:41.167191 10013 raft_consensus.cc:740] T 59f9115b286d44f0b7eac7c74275cb21 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:41.167738 10013 consensus_queue.cc:260] T 59f9115b286d44f0b7eac7c74275cb21 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.169606 10013 ts_tablet_manager.cc:1434] T 59f9115b286d44f0b7eac7c74275cb21 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.056s user 0.006s sys 0.000s
I20260504 13:56:41.171895 11347 leader_election.cc:290] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:41.177755 11355 leader_election.cc:290] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:41.138087 9674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ddf890c17964e7f90409ebe3c4034ef" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:41.178431 9674 raft_consensus.cc:2468] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
W20260504 13:56:41.179734 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 356091us
I20260504 13:56:41.140177 9677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ffeae4e5af4cc8a3eb3bc0bedf18cc" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
W20260504 13:56:41.207959 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 228530us
I20260504 13:56:41.208814 9620 leader_election.cc:304] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
W20260504 13:56:41.210096 9620 leader_election.cc:343] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:41.210413 9620 leader_election.cc:304] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:41.211575 11347 raft_consensus.cc:2804] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:41.211948 11347 raft_consensus.cc:493] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:41.212261 11347 raft_consensus.cc:3060] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:41.214073 11355 raft_consensus.cc:2749] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:41.224785 9620 leader_election.cc:343] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:41.225198 9620 leader_election.cc:304] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:41.226154 9620 leader_election.cc:343] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:41.227746 9768 leader_election.cc:304] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
W20260504 13:56:41.232689 11373 log.cc:927] Time spent T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260504 13:56:41.233224 11383 raft_consensus.cc:2749] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:41.232033 9677 raft_consensus.cc:3060] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:41.235721 11255 raft_consensus.cc:2804] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:41.237087 11380 consensus_queue.cc:1048] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260504 13:56:41.238250 11255 raft_consensus.cc:493] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:41.238562 11255 raft_consensus.cc:3060] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:41.249272 11234 leader_election.cc:290] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:41.249806 9677 raft_consensus.cc:2468] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
W20260504 13:56:41.264001 9618 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 149072us
I20260504 13:56:41.267215 9768 leader_election.cc:304] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:41.268252 11380 raft_consensus.cc:2804] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:41.268688 11380 raft_consensus.cc:697] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:41.269495 11380 consensus_queue.cc:237] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.269838 11206 consensus_queue.cc:1048] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:41.277153 11255 raft_consensus.cc:515] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:41.278307 11347 raft_consensus.cc:515] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.280629 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ddf890c17964e7f90409ebe3c4034ef" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:41.282171 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6323e1c10e7643f8a28f04e285f620e9" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:41.283408 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e29a7365caf04506b15089f10e6c1677" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:41.308588 9618 leader_election.cc:343] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:41.309191 9618 leader_election.cc:343] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:41.309698 9618 leader_election.cc:343] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:41.309885 9932 ts_tablet_manager.cc:1434] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.307s user 0.014s sys 0.009s
I20260504 13:56:41.310909 9932 tablet_bootstrap.cc:492] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:41.313851 11347 leader_election.cc:290] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
W20260504 13:56:41.318480 9175 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 57983fcecdb64a42884f797b4db57df0 consensus state has no leader
I20260504 13:56:41.380128 11255 leader_election.cc:290] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:41.351056 9932 tablet_bootstrap.cc:654] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:41.380978 11412 tablet.cc:1724] T 4a470ddc5bcb4121bd2f5f02f6e6b945 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:41.397665 9931 tablet_bootstrap.cc:492] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:41.398064 9931 ts_tablet_manager.cc:1403] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.535s user 0.012s sys 0.000s
I20260504 13:56:41.411770 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a862a43c5ac443d9bac98e0472ca23b2" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:41.418417 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc2f95dadd544f57bd6d1467d30e954d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
W20260504 13:56:41.423800 9618 leader_election.cc:343] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:41.424465 11381 raft_consensus.cc:515] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:41.226858 9620 leader_election.cc:343] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:41.433611 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 206757us
I20260504 13:56:41.434188 9620 leader_election.cc:304] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:41.436828 11399 raft_consensus.cc:2764] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260504 13:56:41.438169 11399 raft_consensus.cc:493] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:41.438536 11399 raft_consensus.cc:515] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.439898 9620 leader_election.cc:304] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
W20260504 13:56:41.446084 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 111706us
I20260504 13:56:41.444842 11392 raft_consensus.cc:493] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:41.453572 9931 raft_consensus.cc:359] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.454249 9931 raft_consensus.cc:385] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:41.454538 9931 raft_consensus.cc:740] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:41.460597 9189 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 1c12ffa802b840b1937f6f796083c478 (table default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1]). Expected version 1 got 0
I20260504 13:56:41.461388 9189 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 3 got 0
I20260504 13:56:41.463090 9189 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 88619eaa917d4c33a910fc1347ab126e (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:41.453378 11392 raft_consensus.cc:515] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.466528 11333 raft_consensus.cc:2804] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:41.467023 11333 raft_consensus.cc:697] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:41.469017 9189 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4]). Expected version 1 got 0
I20260504 13:56:41.471153 9189 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
I20260504 13:56:41.475632 9831 raft_consensus.cc:3060] T 89edd71cf5aa4a6abe2084695d3ed807 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:41.479861 11392 leader_election.cc:290] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:41.481035 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36c8e9ba29a1466082a3e8bedec87b91" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:41.487823 9189 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38]). Expected version 1 got 0
I20260504 13:56:41.489248 11381 leader_election.cc:290] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:41.493196 9189 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet de6714ee86d64112baf13dd1b2b87582 (table default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469]). Expected version 1 got 0
I20260504 13:56:41.494643 10013 tablet_bootstrap.cc:492] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:41.497028 9189 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 2ddf890c17964e7f90409ebe3c4034ef (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]). Expected version 2 got 0
I20260504 13:56:41.497658 9189 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]). Expected version 1 got 0
I20260504 13:56:41.500581 9530 raft_consensus.cc:3060] T 59f9115b286d44f0b7eac7c74275cb21 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:41.500751 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ddf890c17964e7f90409ebe3c4034ef" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:41.501713 9532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e819494171485e9af8bc8ec33a6ff0" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:41.502753 9533 raft_consensus.cc:3060] T 3d9bd2722d23420ea0eac7664357a60c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:41.503355 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e819494171485e9af8bc8ec33a6ff0" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:41.504340 9537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ffeae4e5af4cc8a3eb3bc0bedf18cc" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:41.503418 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ddf890c17964e7f90409ebe3c4034ef" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:41.499284 9189 catalog_manager.cc:5671] T 538c8c2412104c3e8b19394a1cc38818 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:41.512667 10013 tablet_bootstrap.cc:654] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:41.513224 9189 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet e53381ee0e7f4e9781b833b435bca831 (table default.Table_9b40ba316993413181a9743fb7bab7fb [id=fb4e5ce5526e44de8effdd766f4c257f]). Expected version 1 got 0
I20260504 13:56:41.513809 9189 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]). Expected version 1 got 0
I20260504 13:56:41.514596 9189 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet dffddd4b89984c54bc26dcb4a9488525 (table default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568]). Expected version 1 got 0
I20260504 13:56:41.516402 9187 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: default.Table_42f086bc327c42ccad96925c681a386a [id=2c6aa17153154189a33f5f3aa00b62e9] alter complete (version 1)
I20260504 13:56:41.519682 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ddf890c17964e7f90409ebe3c4034ef" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:41.520364 9831 raft_consensus.cc:2393] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5beceb6317444247b31b44a6581707c6 in current term 1: Already voted for candidate d4051d2bf0bf410eb8acd107560df888 in this term.
I20260504 13:56:41.521906 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6323e1c10e7643f8a28f04e285f620e9" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:41.524417 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e29a7365caf04506b15089f10e6c1677" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:41.528007 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc2f95dadd544f57bd6d1467d30e954d" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:41.528466 9829 raft_consensus.cc:3060] T cc2f95dadd544f57bd6d1467d30e954d P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:41.482290 9618 leader_election.cc:343] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:41.532269 9618 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 50015us
I20260504 13:56:41.536804 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99e819494171485e9af8bc8ec33a6ff0" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:41.537267 9684 raft_consensus.cc:3060] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:41.543087 9829 raft_consensus.cc:2468] T cc2f95dadd544f57bd6d1467d30e954d P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:41.545814 9187 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81] alter complete (version 2)
I20260504 13:56:41.550906 9931 consensus_queue.cc:260] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.552675 9931 ts_tablet_manager.cc:1434] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.154s user 0.004s sys 0.001s
I20260504 13:56:41.569262 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a862a43c5ac443d9bac98e0472ca23b2" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:41.569828 9682 raft_consensus.cc:3060] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:41.577617 11399 leader_election.cc:290] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:41.561631 9678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ddf890c17964e7f90409ebe3c4034ef" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:41.578563 9678 raft_consensus.cc:3060] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:41.480491 11333 consensus_queue.cc:237] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.581183 9188 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
I20260504 13:56:41.582811 9188 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet dffddd4b89984c54bc26dcb4a9488525 (table default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568]). Expected version 1 got 0
I20260504 13:56:41.583127 9684 raft_consensus.cc:2468] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:41.583636 9188 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet de6714ee86d64112baf13dd1b2b87582 (table default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469]). Expected version 1 got 0
I20260504 13:56:41.584023 9768 leader_election.cc:304] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:41.584745 11381 raft_consensus.cc:2804] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:41.585165 11381 raft_consensus.cc:697] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:41.585214 9188 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38]). Expected version 1 got 0
I20260504 13:56:41.585870 11381 consensus_queue.cc:237] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.601011 9932 tablet_bootstrap.cc:492] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:41.601405 9932 ts_tablet_manager.cc:1403] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.291s user 0.011s sys 0.024s
W20260504 13:56:41.602530 9474 leader_election.cc:343] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:41.605667 9682 raft_consensus.cc:2468] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:41.612262 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c09e9076214405eaa8846fe08158bb3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:41.612502 9931 tablet_bootstrap.cc:492] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:41.619678 9618 leader_election.cc:343] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:41.605577 9932 raft_consensus.cc:359] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.608306 11417 tablet.cc:1724] T de6714ee86d64112baf13dd1b2b87582 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:41.623301 9932 raft_consensus.cc:385] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:41.624344 9932 raft_consensus.cc:740] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:41.625730 9932 consensus_queue.cc:260] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.615581 9678 raft_consensus.cc:2468] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:41.627701 9932 ts_tablet_manager.cc:1434] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.026s user 0.004s sys 0.002s
I20260504 13:56:41.627053 9768 leader_election.cc:304] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:41.628520 9932 tablet_bootstrap.cc:492] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:41.628418 11255 raft_consensus.cc:2804] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:41.629135 11255 raft_consensus.cc:697] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:41.629966 11255 consensus_queue.cc:237] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:41.651736 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
I20260504 13:56:41.658254 9932 tablet_bootstrap.cc:654] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:41.666401 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet dffddd4b89984c54bc26dcb4a9488525 (table default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568]). Expected version 1 got 0
I20260504 13:56:41.667297 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38]). Expected version 1 got 0
I20260504 13:56:41.676311 9185 catalog_manager.cc:5671] T 59f9115b286d44f0b7eac7c74275cb21 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:41.683838 10013 tablet_bootstrap.cc:492] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:41.684247 10013 ts_tablet_manager.cc:1403] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.190s user 0.009s sys 0.000s
I20260504 13:56:41.684567 9187 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13] alter complete (version 2)
I20260504 13:56:41.686777 10013 raft_consensus.cc:359] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.687453 10013 raft_consensus.cc:385] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:41.687734 10013 raft_consensus.cc:740] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:41.688376 10013 consensus_queue.cc:260] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:41.690480 9188 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:45997 (request call id 268) took 1389 ms (client timeout 1000 ms). Trace:
W20260504 13:56:41.697167 9188 rpcz_store.cc:269] 0504 13:56:40.300612 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:40.300737 (+ 125us) service_pool.cc:224] Handling call
0504 13:56:41.690388 (+1389651us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":17}
I20260504 13:56:41.695484 9185 catalog_manager.cc:5671] T 3d9bd2722d23420ea0eac7664357a60c P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:41.699736 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet de6714ee86d64112baf13dd1b2b87582 (table default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469]). Expected version 1 got 0
I20260504 13:56:41.703490 9185 catalog_manager.cc:5671] T 982f7c53a1f0482e862b53aeeee7cd25 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:41.706694 9185 catalog_manager.cc:5671] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:41.709712 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet e53381ee0e7f4e9781b833b435bca831 (table default.Table_9b40ba316993413181a9743fb7bab7fb [id=fb4e5ce5526e44de8effdd766f4c257f]). Expected version 1 got 0
I20260504 13:56:41.711349 9185 catalog_manager.cc:5671] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } }
I20260504 13:56:41.714069 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 88619eaa917d4c33a910fc1347ab126e (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
W20260504 13:56:41.726715 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:41.737213 9932 tablet_bootstrap.cc:492] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:41.737638 9932 ts_tablet_manager.cc:1403] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.109s user 0.006s sys 0.012s
I20260504 13:56:41.743201 9932 raft_consensus.cc:359] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.743820 9932 raft_consensus.cc:385] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:41.744310 9932 raft_consensus.cc:740] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:41.745389 9932 consensus_queue.cc:260] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.749336 9932 ts_tablet_manager.cc:1434] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260504 13:56:41.767406 11412 tablet.cc:1724] T dffddd4b89984c54bc26dcb4a9488525 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:41.797857 11333 raft_consensus.cc:493] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:41.798328 11333 raft_consensus.cc:515] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.798710 11395 consensus_queue.cc:1048] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:41.799702 9185 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1] alter complete (version 1)
I20260504 13:56:41.800570 11333 leader_election.cc:290] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:41.801759 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "644170f17f1b45a19677d958b9745e16" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:41.815987 9618 leader_election.cc:343] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:41.825037 9767 leader_election.cc:343] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:41.827070 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36c8e9ba29a1466082a3e8bedec87b91" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:41.833353 10013 ts_tablet_manager.cc:1434] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.149s user 0.006s sys 0.000s
W20260504 13:56:41.841449 9767 consensus_peers.cc:597] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:41.842024 9767 leader_election.cc:343] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:41.842813 9767 leader_election.cc:343] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:41.844062 9767 leader_election.cc:343] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:41.844911 9767 consensus_peers.cc:597] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:41.845674 9767 leader_election.cc:343] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:41.847987 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c09e9076214405eaa8846fe08158bb3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:41.853304 9736 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:41.854290 9932 tablet_bootstrap.cc:492] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:41.862003 11409 consensus_queue.cc:1048] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:41.865309 9932 tablet_bootstrap.cc:654] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:41.875840 11412 tablet.cc:1724] T 88619eaa917d4c33a910fc1347ab126e P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:41.884414 11253 consensus_queue.cc:1048] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:41.894461 9931 tablet_bootstrap.cc:654] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:41.906173 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 96709.2us
I20260504 13:56:41.909643 11274 raft_consensus.cc:493] T 3d9bd2722d23420ea0eac7664357a60c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:41.909502 11426 tablet.cc:1724] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:41.910069 11274 raft_consensus.cc:515] T 3d9bd2722d23420ea0eac7664357a60c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.913213 11274 leader_election.cc:290] T 3d9bd2722d23420ea0eac7664357a60c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:41.931416 9659 tablet_service.cc:1558] Processing DeleteTablet for tablet 21d6358b451d4b8d87a22e30e0780f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:25 UTC) from {username='slave'} at 127.0.0.1:34768
I20260504 13:56:41.936667 10013 tablet_bootstrap.cc:492] T 7d27b04d3f4443c5ade9616d8626d50e P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:41.937064 9473 leader_election.cc:304] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:41.945878 9932 tablet_bootstrap.cc:492] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:41.946943 9932 ts_tablet_manager.cc:1403] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.093s user 0.016s sys 0.008s
I20260504 13:56:41.949056 11274 consensus_queue.cc:1048] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:41.948163 11384 raft_consensus.cc:2804] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:41.950541 11384 raft_consensus.cc:697] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:41.951313 11384 consensus_queue.cc:237] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:41.976388 9931 tablet_bootstrap.cc:492] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:41.976814 9931 ts_tablet_manager.cc:1403] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.364s user 0.005s sys 0.004s
I20260504 13:56:41.979390 9931 raft_consensus.cc:359] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.979995 9931 raft_consensus.cc:385] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:41.980238 9931 raft_consensus.cc:740] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:41.983724 10013 tablet_bootstrap.cc:654] T 7d27b04d3f4443c5ade9616d8626d50e P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:41.991686 9328 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92480654807d495b879576b0195871bd consensus state has no leader
W20260504 13:56:42.003796 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 63056.1us
I20260504 13:56:42.015779 11412 tablet.cc:1724] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:42.012713 9932 raft_consensus.cc:359] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.018486 9932 raft_consensus.cc:385] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:42.018751 9932 raft_consensus.cc:740] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:42.024791 10013 tablet_bootstrap.cc:492] T 7d27b04d3f4443c5ade9616d8626d50e P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:42.025149 10013 ts_tablet_manager.cc:1403] T 7d27b04d3f4443c5ade9616d8626d50e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.089s user 0.006s sys 0.003s
I20260504 13:56:42.024526 9932 consensus_queue.cc:260] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.981140 9931 consensus_queue.cc:260] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:41.985657 9620 leader_election.cc:304] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:42.041975 9931 ts_tablet_manager.cc:1434] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.065s user 0.009s sys 0.000s
I20260504 13:56:42.052716 9932 ts_tablet_manager.cc:1434] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.043s user 0.003s sys 0.001s
I20260504 13:56:42.058477 10013 raft_consensus.cc:359] T 7d27b04d3f4443c5ade9616d8626d50e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.058856 11333 raft_consensus.cc:2749] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:42.059101 10013 raft_consensus.cc:385] T 7d27b04d3f4443c5ade9616d8626d50e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260504 13:56:42.058990 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 73417.8us
I20260504 13:56:42.059381 10013 raft_consensus.cc:740] T 7d27b04d3f4443c5ade9616d8626d50e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
W20260504 13:56:42.059719 9620 leader_election.cc:343] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:42.060060 9620 leader_election.cc:304] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:42.059908 10013 consensus_queue.cc:260] T 7d27b04d3f4443c5ade9616d8626d50e P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.060667 11333 raft_consensus.cc:2749] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:42.060950 9620 leader_election.cc:343] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:42.061317 9620 leader_election.cc:304] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:42.063632 10013 ts_tablet_manager.cc:1434] T 7d27b04d3f4443c5ade9616d8626d50e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.038s user 0.006s sys 0.001s
I20260504 13:56:42.064448 10013 tablet_bootstrap.cc:492] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:42.070709 10013 tablet_bootstrap.cc:654] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:42.097519 9931 tablet_bootstrap.cc:492] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:42.104576 11438 tablet.cc:1724] T e53381ee0e7f4e9781b833b435bca831 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:42.119213 11434 raft_consensus.cc:493] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:42.119745 11434 raft_consensus.cc:515] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.121841 11434 leader_election.cc:290] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:42.128114 9932 tablet_bootstrap.cc:492] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:42.132695 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d9bd2722d23420ea0eac7664357a60c" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:42.133652 11347 raft_consensus.cc:2749] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:42.134133 9932 tablet_bootstrap.cc:654] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:42.135289 11234 raft_consensus.cc:493] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:42.135716 11234 raft_consensus.cc:515] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.136782 10013 tablet_bootstrap.cc:492] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:42.137145 10013 ts_tablet_manager.cc:1403] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.073s user 0.009s sys 0.000s
I20260504 13:56:42.137499 11234 leader_election.cc:290] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:42.137955 11234 raft_consensus.cc:493] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:42.139338 10013 raft_consensus.cc:359] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.139882 10013 raft_consensus.cc:385] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:42.140098 10013 raft_consensus.cc:740] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:42.140609 10013 consensus_queue.cc:260] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.140998 9931 tablet_bootstrap.cc:654] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:42.138350 11234 raft_consensus.cc:515] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.145308 11234 leader_election.cc:290] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:42.149439 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57983fcecdb64a42884f797b4db57df0" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:42.150009 9532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71f444ce9214e3f98eff6ac8ca7dde7" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:42.150350 11381 consensus_queue.cc:1048] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:42.152344 9618 leader_election.cc:343] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:42.153107 9618 leader_election.cc:343] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:42.155375 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 94444.7us
I20260504 13:56:42.155902 10013 ts_tablet_manager.cc:1434] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260504 13:56:42.155958 9620 leader_election.cc:304] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:42.156579 11234 raft_consensus.cc:2804] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:42.157351 11234 raft_consensus.cc:697] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:42.158198 11234 consensus_queue.cc:237] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.181483 11381 raft_consensus.cc:493] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:42.181908 11381 raft_consensus.cc:515] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.184072 11381 leader_election.cc:290] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:42.180588 9932 tablet_bootstrap.cc:492] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:42.184644 9678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71f444ce9214e3f98eff6ac8ca7dde7" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:42.185221 9678 raft_consensus.cc:2468] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:42.186201 9768 leader_election.cc:304] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:42.186904 11381 raft_consensus.cc:2804] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:42.187227 11381 raft_consensus.cc:493] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:42.187525 11381 raft_consensus.cc:3060] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:42.184865 9932 ts_tablet_manager.cc:1403] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.057s user 0.008s sys 0.006s
I20260504 13:56:42.198036 11381 raft_consensus.cc:515] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:42.200946 9474 consensus_peers.cc:597] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:42.202092 9932 raft_consensus.cc:359] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.203981 9932 raft_consensus.cc:385] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:42.204286 9932 raft_consensus.cc:740] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:42.211553 9932 consensus_queue.cc:260] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.221176 9932 ts_tablet_manager.cc:1434] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.024s user 0.003s sys 0.002s
I20260504 13:56:42.222815 10013 tablet_bootstrap.cc:492] T 982f7c53a1f0482e862b53aeeee7cd25 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:42.230499 10013 tablet_bootstrap.cc:654] T 982f7c53a1f0482e862b53aeeee7cd25 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:42.233680 9932 tablet_bootstrap.cc:492] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:42.161648 9620 leader_election.cc:343] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:42.239323 9620 leader_election.cc:304] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:42.240000 11234 raft_consensus.cc:2749] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:42.240125 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 79663.3us
W20260504 13:56:42.240809 9620 consensus_peers.cc:597] T e53381ee0e7f4e9781b833b435bca831 P 5beceb6317444247b31b44a6581707c6 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:42.240906 9932 tablet_bootstrap.cc:654] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:42.241549 9620 leader_election.cc:343] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:42.241686 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71f444ce9214e3f98eff6ac8ca7dde7" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:42.241895 9620 leader_election.cc:304] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:42.241221 11381 leader_election.cc:290] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:42.242323 9686 raft_consensus.cc:3060] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:42.242547 11234 raft_consensus.cc:2749] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:42.243628 11434 consensus_queue.cc:1048] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:42.249609 9686 raft_consensus.cc:2468] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:42.260044 9768 leader_election.cc:304] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:42.260906 11255 raft_consensus.cc:2804] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:42.261324 11255 raft_consensus.cc:697] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:42.262058 11255 consensus_queue.cc:237] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.280521 10013 tablet_bootstrap.cc:492] T 982f7c53a1f0482e862b53aeeee7cd25 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:42.280928 10013 ts_tablet_manager.cc:1403] T 982f7c53a1f0482e862b53aeeee7cd25 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.058s user 0.005s sys 0.005s
W20260504 13:56:42.289044 9474 consensus_peers.cc:597] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:42.299511 9932 tablet_bootstrap.cc:492] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:42.299990 9932 ts_tablet_manager.cc:1403] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.067s user 0.011s sys 0.000s
I20260504 13:56:42.302289 9932 raft_consensus.cc:359] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:42.305320 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:42.305579 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "644170f17f1b45a19677d958b9745e16" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:42.307085 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57983fcecdb64a42884f797b4db57df0" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:42.307780 9174 catalog_manager.cc:4969] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): delete failed for tablet 21d6358b451d4b8d87a22e30e0780f44 because the tablet was not found. No further retry: Not found: Tablet not found: 21d6358b451d4b8d87a22e30e0780f44
I20260504 13:56:42.308350 10013 raft_consensus.cc:359] T 982f7c53a1f0482e862b53aeeee7cd25 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.308952 10013 raft_consensus.cc:385] T 982f7c53a1f0482e862b53aeeee7cd25 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:42.309211 10013 raft_consensus.cc:740] T 982f7c53a1f0482e862b53aeeee7cd25 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:42.309765 10013 consensus_queue.cc:260] T 982f7c53a1f0482e862b53aeeee7cd25 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.316540 9932 raft_consensus.cc:385] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:42.319267 9932 raft_consensus.cc:740] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:42.321233 9931 tablet_bootstrap.cc:492] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:42.321692 9931 ts_tablet_manager.cc:1403] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.224s user 0.011s sys 0.000s
I20260504 13:56:42.320947 9932 consensus_queue.cc:260] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.325425 9931 raft_consensus.cc:359] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:42.326053 9931 raft_consensus.cc:385] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:42.326308 9931 raft_consensus.cc:740] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:42.327970 9932 ts_tablet_manager.cc:1434] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.028s user 0.004s sys 0.001s
I20260504 13:56:42.331291 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71f444ce9214e3f98eff6ac8ca7dde7" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:42.346200 9932 tablet_bootstrap.cc:492] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:42.374914 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d9bd2722d23420ea0eac7664357a60c" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:42.329097 9931 consensus_queue.cc:260] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:42.377837 11381 raft_consensus.cc:493] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:42.378237 11381 raft_consensus.cc:515] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:42.382442 11381 leader_election.cc:290] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:42.382903 11392 raft_consensus.cc:493] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:42.383327 11392 raft_consensus.cc:515] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.385022 11392 leader_election.cc:290] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:42.387423 9686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c13c4cd2f66426f8e7e81eeac7bc0f3" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:42.388053 9686 raft_consensus.cc:2393] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:42.390292 9932 tablet_bootstrap.cc:654] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:42.400017 9931 ts_tablet_manager.cc:1434] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.078s user 0.007s sys 0.000s
I20260504 13:56:42.416723 9931 tablet_bootstrap.cc:492] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:42.421204 9932 tablet_bootstrap.cc:492] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:42.426702 9932 ts_tablet_manager.cc:1403] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.081s user 0.009s sys 0.001s
I20260504 13:56:42.427217 11392 raft_consensus.cc:493] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:42.427670 11392 raft_consensus.cc:515] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.429371 11392 leader_election.cc:290] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:42.451268 9931 tablet_bootstrap.cc:654] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:42.447091 9932 raft_consensus.cc:359] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.453632 9932 raft_consensus.cc:385] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:42.453881 9932 raft_consensus.cc:740] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:42.461405 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b84c3bedc7a84e0d8a873fb15689b314" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:42.462000 9682 raft_consensus.cc:2468] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
W20260504 13:56:42.471766 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 180505us
I20260504 13:56:42.480702 11392 raft_consensus.cc:493] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:42.481207 11392 raft_consensus.cc:515] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.482924 11392 leader_election.cc:290] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:42.491645 9532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f80ce19e8f9498b98952a371476f185" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:42.494951 9932 consensus_queue.cc:260] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.498353 11255 raft_consensus.cc:493] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:42.498808 11255 raft_consensus.cc:515] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.499454 9932 ts_tablet_manager.cc:1434] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.072s user 0.007s sys 0.000s
I20260504 13:56:42.500634 11255 leader_election.cc:290] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:42.502323 9675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36c8e9ba29a1466082a3e8bedec87b91" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:42.502856 9675 raft_consensus.cc:2468] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:42.504243 9768 leader_election.cc:304] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:42.506014 11392 raft_consensus.cc:493] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:42.506584 11392 raft_consensus.cc:515] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.508442 11392 leader_election.cc:290] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:42.509838 11381 raft_consensus.cc:2804] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:42.510187 11381 raft_consensus.cc:493] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:42.510533 11381 raft_consensus.cc:3060] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:42.517653 11381 raft_consensus.cc:515] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.520730 9678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36c8e9ba29a1466082a3e8bedec87b91" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:42.521287 9678 raft_consensus.cc:3060] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:42.529613 9678 raft_consensus.cc:2468] T 36c8e9ba29a1466082a3e8bedec87b91 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:42.530740 9768 leader_election.cc:304] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:42.531513 11255 raft_consensus.cc:2804] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:42.532230 11255 raft_consensus.cc:697] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:42.533349 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddd30e9259a04d59b5c1bb82aabf7952" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:42.533234 11255 consensus_queue.cc:237] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.534706 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd635a863f7a41d087947f619e41f95c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:42.535879 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da61c91e10264fa49fe397d9a9b014bb" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:42.537424 9618 leader_election.cc:343] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:42.536895 11381 leader_election.cc:290] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:42.543375 9581 debug-util.cc:398] Leaking SignalData structure 0x7b0800064940 after lost signal to thread 9446
W20260504 13:56:42.544468 9618 leader_election.cc:343] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:42.545025 9618 leader_election.cc:343] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:42.545943 9581 debug-util.cc:398] Leaking SignalData structure 0x7b080011fe20 after lost signal to thread 9584
I20260504 13:56:42.543110 10013 ts_tablet_manager.cc:1434] T 982f7c53a1f0482e862b53aeeee7cd25 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.262s user 0.002s sys 0.004s
W20260504 13:56:42.558718 9618 leader_election.cc:343] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:42.571882 10013 tablet_bootstrap.cc:492] T 88619eaa917d4c33a910fc1347ab126e P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:42.579191 9532 raft_consensus.cc:3060] T 7d27b04d3f4443c5ade9616d8626d50e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:42.599350 9881 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
W20260504 13:56:42.605230 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 132153us
I20260504 13:56:42.609742 9931 tablet_bootstrap.cc:492] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:42.610241 9931 ts_tablet_manager.cc:1403] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.195s user 0.019s sys 0.011s
I20260504 13:56:42.618557 10013 tablet_bootstrap.cc:654] T 88619eaa917d4c33a910fc1347ab126e P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:42.625558 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd635a863f7a41d087947f619e41f95c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:42.627298 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f80ce19e8f9498b98952a371476f185" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:42.631778 11392 consensus_queue.cc:1048] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:42.640215 9932 tablet_bootstrap.cc:492] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:42.647339 11392 raft_consensus.cc:493] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:42.647781 11392 raft_consensus.cc:515] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.649551 11392 leader_election.cc:290] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:42.650878 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f4c8ffb2175493a981c18676485ac04" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:42.668221 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 49099.6us
W20260504 13:56:42.669354 9474 consensus_peers.cc:597] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:42.671610 9618 leader_election.cc:343] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:42.677083 11381 consensus_queue.cc:1048] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:42.684968 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddd30e9259a04d59b5c1bb82aabf7952" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:42.685523 9620 leader_election.cc:343] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:42.685890 9620 leader_election.cc:304] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:42.685890 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da61c91e10264fa49fe397d9a9b014bb" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:42.686535 11392 raft_consensus.cc:2749] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:42.686825 9620 leader_election.cc:343] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:42.662886 9931 raft_consensus.cc:359] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.691639 9931 raft_consensus.cc:385] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:42.692016 9931 raft_consensus.cc:740] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:42.692783 9931 consensus_queue.cc:260] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.694399 11392 raft_consensus.cc:493] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:42.694847 11392 raft_consensus.cc:515] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.696642 11392 leader_election.cc:290] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:42.698310 9827 raft_consensus.cc:2468] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
W20260504 13:56:42.706034 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 69083.9us
I20260504 13:56:42.709823 9588 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:42.711568 9932 tablet_bootstrap.cc:654] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:42.699873 9931 ts_tablet_manager.cc:1434] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.089s user 0.006s sys 0.000s
I20260504 13:56:42.713968 9931 tablet_bootstrap.cc:492] T e53381ee0e7f4e9781b833b435bca831 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:42.715854 9827 raft_consensus.cc:3060] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:42.715775 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99fe335b131b45a587ec1cf526dd7f79" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:42.724721 9473 leader_election.cc:304] T 3d9bd2722d23420ea0eac7664357a60c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:42.724987 11454 consensus_queue.cc:1048] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:42.726495 11424 consensus_queue.cc:1048] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:42.727494 11433 raft_consensus.cc:2749] T 3d9bd2722d23420ea0eac7664357a60c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:56:42.731926 9618 leader_election.cc:343] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:42.739306 9620 leader_election.cc:304] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:42.740106 11392 raft_consensus.cc:2749] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:42.740208 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 53408us
I20260504 13:56:42.741377 9620 leader_election.cc:304] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:42.742082 11392 raft_consensus.cc:2749] T f71f444ce9214e3f98eff6ac8ca7dde7 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:42.749245 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b84c3bedc7a84e0d8a873fb15689b314" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:42.751219 9473 leader_election.cc:304] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:42.755540 11253 consensus_queue.cc:1048] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:42.810302 9931 tablet_bootstrap.cc:654] T e53381ee0e7f4e9781b833b435bca831 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:42.833130 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:42.845171 11468 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 806407748d564439807fe5d73db9abda)
I20260504 13:56:42.845886 11468 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:42.847882 11468 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
W20260504 13:56:42.851315 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 100207us
I20260504 13:56:42.851963 11424 raft_consensus.cc:2804] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:42.852367 11424 raft_consensus.cc:493] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:42.852780 11424 raft_consensus.cc:3060] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:42.852780 11454 raft_consensus.cc:493] T 982f7c53a1f0482e862b53aeeee7cd25 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:42.855845 11454 raft_consensus.cc:515] T 982f7c53a1f0482e862b53aeeee7cd25 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.861968 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 5 candidate_status { last_received { term: 4 index: 367 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:42.875592 11454 leader_election.cc:290] T 982f7c53a1f0482e862b53aeeee7cd25 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:42.881913 9530 raft_consensus.cc:3060] T 982f7c53a1f0482e862b53aeeee7cd25 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:42.883594 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71f444ce9214e3f98eff6ac8ca7dde7" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:42.899283 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71f444ce9214e3f98eff6ac8ca7dde7" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:42.920192 10013 tablet_bootstrap.cc:492] T 88619eaa917d4c33a910fc1347ab126e P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:42.922628 9932 tablet_bootstrap.cc:492] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:42.923445 10013 ts_tablet_manager.cc:1403] T 88619eaa917d4c33a910fc1347ab126e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.352s user 0.023s sys 0.005s
I20260504 13:56:42.930806 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c13c4cd2f66426f8e7e81eeac7bc0f3" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:42.931041 11424 raft_consensus.cc:515] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.936056 11424 leader_election.cc:290] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:42.926435 11234 raft_consensus.cc:493] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 13:56:42.742780 9620 leader_election.cc:343] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:42.949052 9620 leader_election.cc:304] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:42.949069 11234 raft_consensus.cc:515] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.949708 11392 raft_consensus.cc:2749] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:42.949805 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 207038us
W20260504 13:56:42.950585 9620 leader_election.cc:343] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:42.951097 11234 leader_election.cc:290] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:42.952180 9932 ts_tablet_manager.cc:1403] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.312s user 0.012s sys 0.008s
I20260504 13:56:42.955611 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75a43e11c8b34bd7bcddbf6e9b8f30dc" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:42.955687 9532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36c8e9ba29a1466082a3e8bedec87b91" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:42.950893 9620 leader_election.cc:304] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:42.958720 11234 raft_consensus.cc:2749] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:42.959017 9620 leader_election.cc:343] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:42.959394 9620 leader_election.cc:304] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:42.960044 11234 raft_consensus.cc:2749] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:42.960608 9620 leader_election.cc:304] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
W20260504 13:56:42.963794 9618 leader_election.cc:343] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:42.966794 11392 raft_consensus.cc:2804] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:42.968628 11392 raft_consensus.cc:493] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:42.968988 11392 raft_consensus.cc:3060] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:42.969859 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36c8e9ba29a1466082a3e8bedec87b91" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:42.975939 11392 raft_consensus.cc:515] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.979801 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da61c91e10264fa49fe397d9a9b014bb" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
W20260504 13:56:42.980981 9618 leader_election.cc:343] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:42.980978 9186 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 3 got 0
I20260504 13:56:42.982681 9186 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 88619eaa917d4c33a910fc1347ab126e (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:42.985065 9932 raft_consensus.cc:359] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:42.985687 9932 raft_consensus.cc:385] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:42.985924 9932 raft_consensus.cc:740] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
W20260504 13:56:42.991341 9768 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 116049us
I20260504 13:56:42.998701 11392 leader_election.cc:290] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:43.016009 9931 tablet_bootstrap.cc:492] T e53381ee0e7f4e9781b833b435bca831 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:43.016390 9931 ts_tablet_manager.cc:1403] T e53381ee0e7f4e9781b833b435bca831 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.303s user 0.016s sys 0.002s
I20260504 13:56:43.020092 10013 raft_consensus.cc:359] T 88619eaa917d4c33a910fc1347ab126e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:43.020730 10013 raft_consensus.cc:385] T 88619eaa917d4c33a910fc1347ab126e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:43.020990 10013 raft_consensus.cc:740] T 88619eaa917d4c33a910fc1347ab126e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:43.021497 10013 consensus_queue.cc:260] T 88619eaa917d4c33a910fc1347ab126e P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:43.042110 9736 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:43.053284 9932 consensus_queue.cc:260] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.054594 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "982f7c53a1f0482e862b53aeeee7cd25" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:43.057983 9767 consensus_peers.cc:597] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:43.058511 9767 leader_election.cc:343] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:43.059278 9767 leader_election.cc:343] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:43.060081 9767 consensus_peers.cc:597] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:43.096174 9932 ts_tablet_manager.cc:1434] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.143s user 0.004s sys 0.002s
I20260504 13:56:43.100968 9932 tablet_bootstrap.cc:492] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:43.101814 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "982f7c53a1f0482e862b53aeeee7cd25" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:43.103315 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b84c3bedc7a84e0d8a873fb15689b314" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:43.111779 9767 leader_election.cc:304] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
W20260504 13:56:43.112977 9767 leader_election.cc:343] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.113871 11472 raft_consensus.cc:2749] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:43.114342 11253 consensus_queue.cc:1048] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:43.126999 9186 catalog_manager.cc:5671] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:43.129575 9186 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4]). Expected version 1 got 0
I20260504 13:56:43.018904 9931 raft_consensus.cc:359] T e53381ee0e7f4e9781b833b435bca831 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.134905 9931 raft_consensus.cc:385] T e53381ee0e7f4e9781b833b435bca831 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:43.135260 9931 raft_consensus.cc:740] T e53381ee0e7f4e9781b833b435bca831 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:43.135995 9931 consensus_queue.cc:260] T e53381ee0e7f4e9781b833b435bca831 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.139022 9931 ts_tablet_manager.cc:1434] T e53381ee0e7f4e9781b833b435bca831 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.122s user 0.006s sys 0.000s
I20260504 13:56:43.144827 10013 ts_tablet_manager.cc:1434] T 88619eaa917d4c33a910fc1347ab126e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.210s user 0.004s sys 0.002s
W20260504 13:56:43.114066 9767 leader_election.cc:343] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.172675 9675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b84c3bedc7a84e0d8a873fb15689b314" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:43.173437 9675 raft_consensus.cc:3060] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:43.177475 9767 consensus_peers.cc:597] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:43.183073 9932 tablet_bootstrap.cc:654] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:43.185101 9186 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
I20260504 13:56:43.195451 10013 tablet_bootstrap.cc:492] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:43.198872 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f4c8ffb2175493a981c18676485ac04" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:43.200924 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99fe335b131b45a587ec1cf526dd7f79" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:43.212390 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 50450.7us
W20260504 13:56:43.213379 9474 leader_election.cc:343] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.215950 11432 consensus_queue.cc:1048] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:43.221510 9474 consensus_peers.cc:597] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:43.223760 9931 tablet_bootstrap.cc:492] T 92480654807d495b879576b0195871bd P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:43.229532 10013 tablet_bootstrap.cc:654] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:43.239544 9474 leader_election.cc:343] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.240823 9827 raft_consensus.cc:3060] T e53381ee0e7f4e9781b833b435bca831 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:43.241341 11234 raft_consensus.cc:493] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:43.241782 11234 raft_consensus.cc:515] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.243647 11234 leader_election.cc:290] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:43.250012 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75a43e11c8b34bd7bcddbf6e9b8f30dc" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:43.251531 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6677a93d01d54f15a506ae1474bcb501" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:43.252863 9618 leader_election.cc:343] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.256142 9932 tablet_bootstrap.cc:492] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:43.256635 9932 ts_tablet_manager.cc:1403] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.156s user 0.010s sys 0.000s
I20260504 13:56:43.260962 11454 raft_consensus.cc:493] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:43.261344 11454 raft_consensus.cc:515] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.265061 11454 leader_election.cc:290] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:43.264190 9932 raft_consensus.cc:359] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.265657 9932 raft_consensus.cc:385] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:43.266219 9932 raft_consensus.cc:740] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:43.265684 9531 raft_consensus.cc:3060] T 88619eaa917d4c33a910fc1347ab126e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:43.281450 9675 raft_consensus.cc:2468] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:43.287896 11476 raft_consensus.cc:493] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:43.288457 11476 raft_consensus.cc:515] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:43.292196 9768 leader_election.cc:343] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:43.292757 11476 leader_election.cc:290] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:43.294148 9767 consensus_peers.cc:597] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:43.299716 9932 consensus_queue.cc:260] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.302770 9932 ts_tablet_manager.cc:1434] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.046s user 0.005s sys 0.000s
I20260504 13:56:43.302358 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da61c91e10264fa49fe397d9a9b014bb" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
W20260504 13:56:43.303421 9618 consensus_peers.cc:597] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:43.303604 9829 raft_consensus.cc:3060] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:43.315430 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92480654807d495b879576b0195871bd" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:43.340934 11480 consensus_queue.cc:1048] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:43.341388 9829 raft_consensus.cc:2468] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:43.317827 9932 tablet_bootstrap.cc:492] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:43.362139 9473 leader_election.cc:304] T 982f7c53a1f0482e862b53aeeee7cd25 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:43.362828 11433 raft_consensus.cc:2749] T 982f7c53a1f0482e862b53aeeee7cd25 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:43.378360 9175 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 57983fcecdb64a42884f797b4db57df0 consensus state has no leader
W20260504 13:56:43.388433 11483 log.cc:927] Time spent T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P d4051d2bf0bf410eb8acd107560df888: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260504 13:56:43.394819 11392 raft_consensus.cc:493] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:43.395344 11392 raft_consensus.cc:515] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.397261 11392 leader_election.cc:290] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:43.399983 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37b1664312d347288680d97b62c2aecb" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:43.401234 11234 raft_consensus.cc:493] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260504 13:56:43.402441 9474 leader_election.cc:343] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.401628 11234 raft_consensus.cc:515] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:43.403383 9618 leader_election.cc:343] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.405494 11234 leader_election.cc:290] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:43.407096 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e1d998c4314db3b3682a0510393749" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:43.408370 9618 leader_election.cc:343] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.409978 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6677a93d01d54f15a506ae1474bcb501" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:43.413712 9932 tablet_bootstrap.cc:654] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:43.399017 9186 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 2ddf890c17964e7f90409ebe3c4034ef (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]). Expected version 2 got 0
I20260504 13:56:43.414995 11490 tablet.cc:1724] T e53381ee0e7f4e9781b833b435bca831 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:43.421504 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37b1664312d347288680d97b62c2aecb" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:43.425293 9931 tablet_bootstrap.cc:654] T 92480654807d495b879576b0195871bd P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:43.414242 9186 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]). Expected version 1 got 0
I20260504 13:56:43.428097 9186 catalog_manager.cc:5671] T 7a89bac060214d30924398007dfa1fa1 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:43.430125 9186 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet e53381ee0e7f4e9781b833b435bca831 (table default.Table_9b40ba316993413181a9743fb7bab7fb [id=fb4e5ce5526e44de8effdd766f4c257f]). Expected version 1 got 0
I20260504 13:56:43.434268 10013 tablet_bootstrap.cc:492] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:43.434725 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92480654807d495b879576b0195871bd" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:43.435310 9684 raft_consensus.cc:2468] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
W20260504 13:56:43.452051 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 48867us
W20260504 13:56:43.457999 9620 leader_election.cc:343] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.431084 9186 catalog_manager.cc:5671] T b6103d953ace4cb8be3c580b3316f87c P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:43.460481 9620 leader_election.cc:304] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:43.462544 9186 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]). Expected version 1 got 0
I20260504 13:56:43.460562 11234 raft_consensus.cc:493] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:43.464207 11234 raft_consensus.cc:515] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:43.464605 9620 leader_election.cc:343] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.464954 9620 leader_election.cc:304] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:43.465994 11234 leader_election.cc:290] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:43.466459 11234 raft_consensus.cc:2749] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:43.469395 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df71dc4f2cf84cda8cfc534a0eea86f8" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:43.470187 9767 leader_election.cc:343] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Not found: Tablet not found: 41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:43.470606 9767 leader_election.cc:304] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:43.466933 11234 raft_consensus.cc:2749] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:43.467813 11347 consensus_queue.cc:1048] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:43.471247 11253 raft_consensus.cc:2749] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:56:43.470691 9618 leader_election.cc:343] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.472565 9473 leader_election.cc:304] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:43.473330 11274 raft_consensus.cc:2804] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:43.473774 11274 raft_consensus.cc:697] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:43.474524 11274 consensus_queue.cc:237] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:43.478426 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:43.495586 11487 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:43.496589 11487 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:43.439042 10013 ts_tablet_manager.cc:1403] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.249s user 0.004s sys 0.011s
I20260504 13:56:43.525801 10013 raft_consensus.cc:359] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.526530 10013 raft_consensus.cc:385] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:43.527508 10013 raft_consensus.cc:740] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:43.528066 11253 raft_consensus.cc:493] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:43.528620 11253 raft_consensus.cc:515] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:43.532388 9768 leader_election.cc:343] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 8bb77106b48a45a2aaa938dc474fa8bd
I20260504 13:56:43.532955 11253 leader_election.cc:290] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:43.539633 11487 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:56:43.544854 9932 tablet_bootstrap.cc:492] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:43.545372 9932 ts_tablet_manager.cc:1403] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.230s user 0.013s sys 0.017s
I20260504 13:56:43.548863 9932 raft_consensus.cc:359] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.549533 9932 raft_consensus.cc:385] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:43.549844 9932 raft_consensus.cc:740] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:43.550429 9932 consensus_queue.cc:260] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.552913 9932 ts_tablet_manager.cc:1434] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260504 13:56:43.555820 9767 leader_election.cc:343] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Not found: Tablet not found: 8bb77106b48a45a2aaa938dc474fa8bd
I20260504 13:56:43.556219 9767 leader_election.cc:304] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:43.557157 11253 raft_consensus.cc:2749] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:43.573158 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_2f0fc59ee89c402cb8a0fc23341636e6 [id=00b5922717934e15993bafde9f2513e7]
I20260504 13:56:43.591641 10013 consensus_queue.cc:260] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.535501 11497 tablet.cc:1724] T 88619eaa917d4c33a910fc1347ab126e P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260504 13:56:43.600270 10690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.252:44781, user_credentials={real_user=slave}} blocked reactor thread for 124505us
I20260504 13:56:43.603771 9932 tablet_bootstrap.cc:492] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:43.502491 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:43.614569 9931 tablet_bootstrap.cc:492] T 92480654807d495b879576b0195871bd P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:43.615041 9931 ts_tablet_manager.cc:1403] T 92480654807d495b879576b0195871bd P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.391s user 0.008s sys 0.002s
I20260504 13:56:43.618461 10013 ts_tablet_manager.cc:1434] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.095s user 0.004s sys 0.000s
I20260504 13:56:43.619613 10013 tablet_bootstrap.cc:492] T 99e819494171485e9af8bc8ec33a6ff0 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:43.621156 9931 raft_consensus.cc:359] T 92480654807d495b879576b0195871bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.621855 9931 raft_consensus.cc:385] T 92480654807d495b879576b0195871bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260504 13:56:43.621758 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 155947us
I20260504 13:56:43.622172 9931 raft_consensus.cc:740] T 92480654807d495b879576b0195871bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
W20260504 13:56:43.622308 9620 leader_election.cc:343] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.622637 9620 leader_election.cc:304] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:43.623826 11234 raft_consensus.cc:2749] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:43.624058 9620 leader_election.cc:304] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:43.624763 11234 raft_consensus.cc:2804] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
W20260504 13:56:43.624903 9620 leader_election.cc:343] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.625165 11234 raft_consensus.cc:697] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:43.625278 9620 leader_election.cc:304] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:43.625917 11234 consensus_queue.cc:237] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.627880 11392 raft_consensus.cc:2749] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:43.628911 10810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:43.632094 9328 leader_election.cc:304] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec9f4c3a43d84fc9a420dd10723c01a1; no voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda
I20260504 13:56:43.632884 11487 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 13:56:43.643306 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_798d2540b8f4489fb11763ccaa36bced [id=049b4fa786fc4dbb8e52931dcaaab90f]
I20260504 13:56:43.646575 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_512ae07c2df74cb99bb77006d985f2da [id=09959f81c69c49778a6ee33dc1e85aae]
W20260504 13:56:43.682725 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 57838.3us
W20260504 13:56:43.683542 9620 leader_election.cc:343] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.641770 11255 consensus_queue.cc:1048] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:43.683900 9620 leader_election.cc:304] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:43.686376 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e1d998c4314db3b3682a0510393749" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:43.687285 11234 raft_consensus.cc:2749] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:43.688141 9473 leader_election.cc:304] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:43.688072 9931 consensus_queue.cc:260] T 92480654807d495b879576b0195871bd P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.689893 11384 raft_consensus.cc:2804] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:43.690207 11384 raft_consensus.cc:493] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:43.690295 9931 ts_tablet_manager.cc:1434] T 92480654807d495b879576b0195871bd P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.075s user 0.004s sys 0.000s
I20260504 13:56:43.690486 11384 raft_consensus.cc:3060] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:43.692328 9932 tablet_bootstrap.cc:654] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:43.702601 10013 tablet_bootstrap.cc:654] T 99e819494171485e9af8bc8ec33a6ff0 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:43.707425 11384 raft_consensus.cc:515] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.710331 11384 leader_election.cc:290] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:43.721239 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92480654807d495b879576b0195871bd" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:43.721798 9834 raft_consensus.cc:3060] T 92480654807d495b879576b0195871bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:43.727814 9931 tablet_bootstrap.cc:492] T 7a89bac060214d30924398007dfa1fa1 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:43.734093 9834 raft_consensus.cc:2468] T 92480654807d495b879576b0195871bd P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:43.735316 9474 leader_election.cc:304] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:43.736089 11274 raft_consensus.cc:2804] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:43.736713 11274 raft_consensus.cc:697] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:43.738343 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38]
I20260504 13:56:43.737515 11274 consensus_queue.cc:237] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.742834 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df71dc4f2cf84cda8cfc534a0eea86f8" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:43.757140 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_dd11ea89ad7248c39ec114af58cf598f [id=14bd1983ba95468e9e70d8f5724f42e6]
I20260504 13:56:43.775782 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_2bed5ea8f43149e8bc3b21cd9d1b3846 [id=15994ccc978545058e324152f70c8c3e]
W20260504 13:56:43.779318 9767 consensus_peers.cc:597] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:43.780670 9931 tablet_bootstrap.cc:654] T 7a89bac060214d30924398007dfa1fa1 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:43.784356 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_dee1335eff89438e9c68835a4c258031 [id=1714df2b31ff4865b6b357500d121a26]
I20260504 13:56:43.808460 10013 tablet_bootstrap.cc:492] T 99e819494171485e9af8bc8ec33a6ff0 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:43.808964 10013 ts_tablet_manager.cc:1403] T 99e819494171485e9af8bc8ec33a6ff0 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.190s user 0.007s sys 0.003s
I20260504 13:56:43.811614 10013 raft_consensus.cc:359] T 99e819494171485e9af8bc8ec33a6ff0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.818017 10013 raft_consensus.cc:385] T 99e819494171485e9af8bc8ec33a6ff0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:43.819555 10013 raft_consensus.cc:740] T 99e819494171485e9af8bc8ec33a6ff0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:43.821506 9931 tablet_bootstrap.cc:492] T 7a89bac060214d30924398007dfa1fa1 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:43.824610 9678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92480654807d495b879576b0195871bd" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:43.825623 9678 raft_consensus.cc:3060] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:43.831980 9931 ts_tablet_manager.cc:1403] T 7a89bac060214d30924398007dfa1fa1 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.104s user 0.010s sys 0.001s
I20260504 13:56:43.827572 10013 consensus_queue.cc:260] T 99e819494171485e9af8bc8ec33a6ff0 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.834339 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_24398dd8ee63413ebb585587353ea155 [id=1d2bbdaf6382433cb8c2a55f62f0388d]
I20260504 13:56:43.834910 10013 ts_tablet_manager.cc:1434] T 99e819494171485e9af8bc8ec33a6ff0 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20260504 13:56:43.841348 9881 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:43.839887 9931 raft_consensus.cc:359] T 7a89bac060214d30924398007dfa1fa1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.844345 9931 raft_consensus.cc:385] T 7a89bac060214d30924398007dfa1fa1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:43.844614 9931 raft_consensus.cc:740] T 7a89bac060214d30924398007dfa1fa1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:43.859932 9932 tablet_bootstrap.cc:492] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:43.860311 9932 ts_tablet_manager.cc:1403] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.257s user 0.006s sys 0.009s
I20260504 13:56:43.859874 9931 consensus_queue.cc:260] T 7a89bac060214d30924398007dfa1fa1 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.864686 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_60fe6ee7a2414e2aa8ea2d57ee4a827d [id=1de44a1bbf06492084cfd70cf540b213]
I20260504 13:56:43.866948 9931 ts_tablet_manager.cc:1434] T 7a89bac060214d30924398007dfa1fa1 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.035s user 0.003s sys 0.004s
I20260504 13:56:43.868422 10013 tablet_bootstrap.cc:492] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:43.869797 11476 raft_consensus.cc:493] T 7a89bac060214d30924398007dfa1fa1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:43.870227 11476 raft_consensus.cc:515] T 7a89bac060214d30924398007dfa1fa1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.872011 11476 leader_election.cc:290] T 7a89bac060214d30924398007dfa1fa1 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:43.872882 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a89bac060214d30924398007dfa1fa1" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:43.875720 9931 tablet_bootstrap.cc:492] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:43.876405 10013 tablet_bootstrap.cc:654] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:43.903559 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a89bac060214d30924398007dfa1fa1" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:43.905146 9932 raft_consensus.cc:359] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.905781 9932 raft_consensus.cc:385] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:43.906021 9932 raft_consensus.cc:740] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:43.906594 9932 consensus_queue.cc:260] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:43.909616 9768 leader_election.cc:304] T 7a89bac060214d30924398007dfa1fa1 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:43.911217 11253 raft_consensus.cc:2749] T 7a89bac060214d30924398007dfa1fa1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:43.919198 9932 ts_tablet_manager.cc:1434] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.059s user 0.003s sys 0.003s
I20260504 13:56:43.919965 9932 tablet_bootstrap.cc:492] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:43.914216 9678 raft_consensus.cc:2468] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
W20260504 13:56:43.927804 9620 leader_election.cc:343] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.928237 9620 leader_election.cc:304] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:43.929082 11234 raft_consensus.cc:2749] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:43.929955 11234 consensus_queue.cc:1048] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:43.935303 9931 tablet_bootstrap.cc:654] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:43.866933 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]
I20260504 13:56:43.964754 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_5ff6dab0e47d41e681533fca02a36153 [id=230a8885dd2b4995815b650d3d9f9f75]
I20260504 13:56:43.967639 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_09255abca80f41dc895d81ccd89b9c7a [id=24485cf365ad4f9b89fa0f716fdb34cd]
I20260504 13:56:43.972234 9932 tablet_bootstrap.cc:654] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:43.976064 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 46050.5us
W20260504 13:56:43.976647 9620 leader_election.cc:343] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:43.977001 9620 leader_election.cc:304] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:43.984648 9931 tablet_bootstrap.cc:492] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:43.986295 9931 ts_tablet_manager.cc:1403] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.111s user 0.007s sys 0.011s
I20260504 13:56:43.989502 9931 raft_consensus.cc:359] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:43.990312 9931 raft_consensus.cc:385] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:43.990607 9931 raft_consensus.cc:740] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:44.002959 9931 consensus_queue.cc:260] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:44.011718 9931 ts_tablet_manager.cc:1434] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20260504 13:56:44.012564 11494 raft_consensus.cc:2749] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:44.020896 10013 tablet_bootstrap.cc:492] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:44.021000 9931 tablet_bootstrap.cc:492] T b6103d953ace4cb8be3c580b3316f87c P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:44.021396 10013 ts_tablet_manager.cc:1403] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.153s user 0.010s sys 0.001s
I20260504 13:56:44.023867 10013 raft_consensus.cc:359] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.028888 9931 tablet_bootstrap.cc:654] T b6103d953ace4cb8be3c580b3316f87c P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:44.036345 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee]
I20260504 13:56:44.037145 11494 raft_consensus.cc:493] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.037549 11494 raft_consensus.cc:515] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:44.039354 11494 leader_election.cc:290] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:44.041103 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_2585508fc4214c8b82d2957644309ac2 [id=256b605e5a2342ed883ec30e3de0ffda]
I20260504 13:56:44.051882 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dd139c347f7406aac340cc6ac994273" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:44.068671 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:44.080240 10013 raft_consensus.cc:385] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:44.080500 10013 raft_consensus.cc:740] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:44.081033 10013 consensus_queue.cc:260] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.040025 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dd139c347f7406aac340cc6ac994273" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:44.064944 9531 raft_consensus.cc:3060] T 99e819494171485e9af8bc8ec33a6ff0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:44.087775 9618 leader_election.cc:343] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.097095 9932 tablet_bootstrap.cc:492] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:44.097590 9932 ts_tablet_manager.cc:1403] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.178s user 0.011s sys 0.000s
I20260504 13:56:44.098817 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_42f086bc327c42ccad96925c681a386a [id=2c6aa17153154189a33f5f3aa00b62e9]
I20260504 13:56:44.100215 9932 raft_consensus.cc:359] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.100798 9932 raft_consensus.cc:385] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:44.101531 9932 raft_consensus.cc:740] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:44.102068 9932 consensus_queue.cc:260] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.106443 9932 ts_tablet_manager.cc:1434] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260504 13:56:44.109997 11234 raft_consensus.cc:493] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.110457 11234 raft_consensus.cc:515] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.112231 11234 leader_election.cc:290] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:44.113672 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7989338bc4bf4bad8fb8fe37587d0a16" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:44.115392 9618 leader_election.cc:343] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.116226 10013 ts_tablet_manager.cc:1434] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.095s user 0.005s sys 0.000s
I20260504 13:56:44.121138 11480 raft_consensus.cc:493] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.132236 11480 raft_consensus.cc:515] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.134104 11480 leader_election.cc:290] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:44.136123 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1a01231bb044433ae5827d07a6592f8" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:44.137229 9474 leader_election.cc:343] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.137760 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_69c9551ae2cf4ef399e40017a33bad73 [id=2d9b8dca85e9477eb77877d5618e9b6f]
I20260504 13:56:44.143425 11480 raft_consensus.cc:493] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.143886 11480 raft_consensus.cc:515] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.145830 9932 tablet_bootstrap.cc:492] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:44.146135 11480 leader_election.cc:290] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:44.146721 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2bfb681e6824e9f9bdcbea23061d57a" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:44.149196 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7989338bc4bf4bad8fb8fe37587d0a16" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:44.153447 9474 leader_election.cc:343] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:44.150727 9620 leader_election.cc:343] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.161969 9620 leader_election.cc:304] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:44.160976 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e]
I20260504 13:56:44.167279 11502 raft_consensus.cc:2749] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:44.170554 10013 tablet_bootstrap.cc:492] T af5c5a6cae4e4b6ea3eece2e21a587ec P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:44.177059 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_1611aa9f4dba40e09dae4433e2d20154 [id=2fb2c56e2fb44b9aac476e06e36e5778]
I20260504 13:56:44.181504 10013 tablet_bootstrap.cc:654] T af5c5a6cae4e4b6ea3eece2e21a587ec P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:44.189044 9931 tablet_bootstrap.cc:492] T b6103d953ace4cb8be3c580b3316f87c P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:44.189409 9931 ts_tablet_manager.cc:1403] T b6103d953ace4cb8be3c580b3316f87c P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.169s user 0.003s sys 0.005s
I20260504 13:56:44.195561 9931 raft_consensus.cc:359] T b6103d953ace4cb8be3c580b3316f87c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.196164 9931 raft_consensus.cc:385] T b6103d953ace4cb8be3c580b3316f87c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:44.196408 9931 raft_consensus.cc:740] T b6103d953ace4cb8be3c580b3316f87c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:44.196945 9931 consensus_queue.cc:260] T b6103d953ace4cb8be3c580b3316f87c P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.198302 9931 ts_tablet_manager.cc:1434] T b6103d953ace4cb8be3c580b3316f87c P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
W20260504 13:56:44.202708 9620 leader_election.cc:343] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.203094 9620 leader_election.cc:304] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:44.203763 11502 raft_consensus.cc:2749] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:44.189051 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_2d5a582adb634895a55193dd861f3b60 [id=30f349197d564e8da82855fec17b4cd2]
I20260504 13:56:44.208813 11502 raft_consensus.cc:493] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.209192 11502 raft_consensus.cc:515] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.210865 11502 leader_election.cc:290] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:44.211611 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564d0fafb49a4b4bb3bdb240c74c3410" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:44.212924 9618 leader_election.cc:343] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.213598 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]
I20260504 13:56:44.215982 9932 tablet_bootstrap.cc:654] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:44.221589 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f]
I20260504 13:56:44.223611 11234 raft_consensus.cc:493] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.224098 11234 raft_consensus.cc:515] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.227797 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2bfb681e6824e9f9bdcbea23061d57a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:44.228356 9531 raft_consensus.cc:2468] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:44.229233 9678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1a01231bb044433ae5827d07a6592f8" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:44.229735 9678 raft_consensus.cc:2468] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:44.229969 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564d0fafb49a4b4bb3bdb240c74c3410" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:44.230441 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2bfb681e6824e9f9bdcbea23061d57a" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:44.230963 9682 raft_consensus.cc:2468] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:44.232607 9618 leader_election.cc:304] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:44.233280 11502 raft_consensus.cc:2804] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:44.233637 11502 raft_consensus.cc:493] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:44.233989 11502 raft_consensus.cc:3060] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:44.240753 11234 leader_election.cc:290] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:44.245028 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_963c93d5d8ed49e6a2052fa1e652da91 [id=399a65bb2fa6428296341de74e964518]
I20260504 13:56:44.245949 9932 tablet_bootstrap.cc:492] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:44.245615 11502 raft_consensus.cc:515] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.246519 9932 ts_tablet_manager.cc:1403] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.101s user 0.009s sys 0.000s
W20260504 13:56:44.246860 10690 leader_election.cc:336] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781): Timed out: RequestConsensusVote RPC to 127.8.238.252:44781 timed out after 0.616s (SENT)
I20260504 13:56:44.247300 10690 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:44.249922 9932 raft_consensus.cc:359] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.259652 10013 tablet_bootstrap.cc:492] T af5c5a6cae4e4b6ea3eece2e21a587ec P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:44.259651 9932 raft_consensus.cc:385] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:44.260120 10013 ts_tablet_manager.cc:1403] T af5c5a6cae4e4b6ea3eece2e21a587ec P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.090s user 0.010s sys 0.000s
W20260504 13:56:44.259691 9620 leader_election.cc:343] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.260491 9620 leader_election.cc:304] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:44.261622 11234 raft_consensus.cc:2749] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:44.263597 9473 leader_election.cc:304] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:44.264660 11434 raft_consensus.cc:2804] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:44.265077 11434 raft_consensus.cc:493] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:44.265410 11434 raft_consensus.cc:3060] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:44.266371 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_f8a23a8252f84cacb13b338f9d9dc9fc [id=39be2c4d16aa41c0832bfeb23269243a]
I20260504 13:56:44.267689 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_073a6dae9da444bea1f96b2f33fe1a85 [id=39c692c399dd43ca963357b6818871cf]
I20260504 13:56:44.264605 10013 raft_consensus.cc:359] T af5c5a6cae4e4b6ea3eece2e21a587ec P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:44.271476 10013 raft_consensus.cc:385] T af5c5a6cae4e4b6ea3eece2e21a587ec P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:44.271781 10013 raft_consensus.cc:740] T af5c5a6cae4e4b6ea3eece2e21a587ec P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:44.272444 10013 consensus_queue.cc:260] T af5c5a6cae4e4b6ea3eece2e21a587ec P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:44.274350 10013 ts_tablet_manager.cc:1434] T af5c5a6cae4e4b6ea3eece2e21a587ec P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20260504 13:56:44.276363 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_4207c9e8019c4088b934376c5568d101 [id=3ca89308de104a5fa27cac4c62047304]
I20260504 13:56:44.277575 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2bfb681e6824e9f9bdcbea23061d57a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:44.274480 11434 raft_consensus.cc:515] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.259904 9932 raft_consensus.cc:740] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:44.281069 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1a01231bb044433ae5827d07a6592f8" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:44.280964 9932 consensus_queue.cc:260] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:44.282024 9474 leader_election.cc:343] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.282617 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]
I20260504 13:56:44.278697 10013 tablet_bootstrap.cc:492] T cc2f95dadd544f57bd6d1467d30e954d P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:44.283670 9932 ts_tablet_manager.cc:1434] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.037s user 0.004s sys 0.001s
I20260504 13:56:44.284524 9932 tablet_bootstrap.cc:492] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:44.286288 11502 leader_election.cc:290] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:44.287186 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_5944297f8c104978bbc6913ecd8f47a4 [id=3f11131cece44f3ebf3cd0f1e69139ae]
I20260504 13:56:44.288499 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]
I20260504 13:56:44.289110 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2bfb681e6824e9f9bdcbea23061d57a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:44.290524 9531 raft_consensus.cc:3060] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:44.289371 11434 leader_election.cc:290] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:44.296460 9932 tablet_bootstrap.cc:654] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:44.300007 9931 tablet_bootstrap.cc:492] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:44.311453 11502 raft_consensus.cc:493] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.312345 11502 raft_consensus.cc:515] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.318311 9932 tablet_bootstrap.cc:492] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:44.318710 9932 ts_tablet_manager.cc:1403] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.006s
I20260504 13:56:44.320417 11524 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 13:56:44.321089 9932 raft_consensus.cc:359] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.321657 9932 raft_consensus.cc:385] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:44.321883 9932 raft_consensus.cc:740] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:44.322372 9932 consensus_queue.cc:260] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.327688 9473 leader_election.cc:304] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:44.329959 9932 ts_tablet_manager.cc:1434] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260504 13:56:44.330729 9932 tablet_bootstrap.cc:492] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:44.331494 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2bfb681e6824e9f9bdcbea23061d57a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:44.336766 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1a01231bb044433ae5827d07a6592f8" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:44.337316 9682 raft_consensus.cc:3060] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:44.338057 10690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.252:44781, user_credentials={real_user=slave}} blocked reactor thread for 91228.3us
I20260504 13:56:44.340879 10013 tablet_bootstrap.cc:654] T cc2f95dadd544f57bd6d1467d30e954d P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:44.313282 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_1665799e268a496d9ffc5568fa34a763 [id=403cbbc8a1304e5695a89d3697932c26]
I20260504 13:56:44.342737 9932 tablet_bootstrap.cc:654] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:44.345914 9682 raft_consensus.cc:2468] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:44.357546 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_003cbc6fdf0b4dddaba008af167dc621 [id=4629003a2d8d461c9e309d49ddecea6a]
I20260504 13:56:44.361028 9932 tablet_bootstrap.cc:492] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:44.361680 9932 ts_tablet_manager.cc:1403] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.005s
I20260504 13:56:44.306954 9531 raft_consensus.cc:2468] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:44.363130 11434 raft_consensus.cc:2764] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260504 13:56:44.364356 9932 raft_consensus.cc:359] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.365134 9932 raft_consensus.cc:385] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:44.365274 9618 leader_election.cc:304] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:44.366003 11234 raft_consensus.cc:2804] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:44.365437 9932 raft_consensus.cc:740] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:44.366464 11234 raft_consensus.cc:697] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:44.359083 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_a660204a2583425aa3ecd09c5cc01368 [id=4683a87561f14c049c1ed8eb68dc24d1]
I20260504 13:56:44.366822 9932 consensus_queue.cc:260] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.369222 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_821e9f466c974a2fbf24c07f8c266e62 [id=469bd5cb4e7f4a8a9c7af947a834ca7b]
I20260504 13:56:44.372560 11434 consensus_queue.cc:1048] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:44.374270 9473 leader_election.cc:304] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:44.375401 11434 raft_consensus.cc:2804] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:44.376397 11434 raft_consensus.cc:697] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:44.377620 9834 raft_consensus.cc:3060] T b6103d953ace4cb8be3c580b3316f87c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:44.377449 11434 consensus_queue.cc:237] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.367198 11234 consensus_queue.cc:237] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.373065 11494 raft_consensus.cc:493] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.381788 11494 raft_consensus.cc:515] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.383997 11494 leader_election.cc:290] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:44.385306 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e70772b7ce4f27a941918aaf66a865" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:44.386646 9618 leader_election.cc:343] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.377044 10013 tablet_bootstrap.cc:492] T cc2f95dadd544f57bd6d1467d30e954d P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:44.387538 10013 ts_tablet_manager.cc:1403] T cc2f95dadd544f57bd6d1467d30e954d P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.109s user 0.010s sys 0.000s
I20260504 13:56:44.392920 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef975650610423c88ce8e4cfb0b013b" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:44.394125 9618 leader_election.cc:343] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:44.394853 9474 consensus_peers.cc:597] T b84c3bedc7a84e0d8a873fb15689b314 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Couldn't send request to peer d4051d2bf0bf410eb8acd107560df888. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:44.396521 11502 leader_election.cc:290] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
W20260504 13:56:44.373844 9620 leader_election.cc:343] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.371781 9736 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
W20260504 13:56:44.403914 9620 leader_election.cc:343] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.405946 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e70772b7ce4f27a941918aaf66a865" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:44.390329 10013 raft_consensus.cc:359] T cc2f95dadd544f57bd6d1467d30e954d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.414914 9931 tablet_bootstrap.cc:654] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:44.415135 10013 raft_consensus.cc:385] T cc2f95dadd544f57bd6d1467d30e954d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:44.415488 10013 raft_consensus.cc:740] T cc2f95dadd544f57bd6d1467d30e954d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:44.416082 10013 consensus_queue.cc:260] T cc2f95dadd544f57bd6d1467d30e954d P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.417572 10013 ts_tablet_manager.cc:1434] T cc2f95dadd544f57bd6d1467d30e954d P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.030s user 0.002s sys 0.002s
I20260504 13:56:44.420888 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef975650610423c88ce8e4cfb0b013b" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:44.424614 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_f7b06aadc0024855b08b56cad8c7b6a3 [id=4efd9dd69847446d9f05700d3bd3f914]
I20260504 13:56:44.426823 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]
I20260504 13:56:44.449383 10013 tablet_bootstrap.cc:492] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:44.452437 11255 raft_consensus.cc:493] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.452930 11255 raft_consensus.cc:515] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:44.457312 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_2d928cb3553f40bc8e825d0741ef3199 [id=51b254be98ee457d892039784e4a5f0b]
I20260504 13:56:44.454929 11255 leader_election.cc:290] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:44.459766 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af5c5a6cae4e4b6ea3eece2e21a587ec" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:44.460317 9531 raft_consensus.cc:2468] T af5c5a6cae4e4b6ea3eece2e21a587ec P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:44.460734 11234 raft_consensus.cc:493] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.461644 9767 leader_election.cc:304] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:44.461427 11234 raft_consensus.cc:515] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.462689 11255 raft_consensus.cc:2804] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:44.463040 11255 raft_consensus.cc:493] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:44.463406 11255 raft_consensus.cc:3060] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:44.464864 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:44.479846 10013 tablet_bootstrap.cc:654] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:44.480716 11234 leader_election.cc:290] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:44.481581 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9e66dacf5794b6195575556be889bce" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:44.483016 9618 leader_election.cc:343] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:44.485746 9618 consensus_peers.cc:597] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:44.463006 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]
I20260504 13:56:44.487960 9678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af5c5a6cae4e4b6ea3eece2e21a587ec" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:44.488647 9678 raft_consensus.cc:2468] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:44.489374 11532 raft_consensus.cc:493] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.489886 11532 raft_consensus.cc:515] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.492192 11532 leader_election.cc:290] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:44.492766 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f337e9698fe64282a684e90d9c61da5a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:44.493858 9620 leader_election.cc:343] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:44.494081 9618 leader_election.cc:343] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.494760 9620 leader_election.cc:304] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:44.496574 9620 leader_election.cc:343] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.505419 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_af058f50341e4f478c970f8cb52bf0d7 [id=57ccd7218d454c8c908fe06b798bd83b]
I20260504 13:56:44.508824 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_597b4ca3e37849d0aa9c86eed64d0614 [id=5a597e1cae1d4126ae355289aef1ced6]
I20260504 13:56:44.512305 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62]
I20260504 13:56:44.515851 11255 raft_consensus.cc:515] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:44.520048 11255 leader_election.cc:290] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:44.505403 9932 ts_tablet_manager.cc:1434] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.143s user 0.015s sys 0.000s
I20260504 13:56:44.503999 11234 raft_consensus.cc:2749] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:44.525899 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af5c5a6cae4e4b6ea3eece2e21a587ec" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:44.526021 9531 raft_consensus.cc:3060] T cc2f95dadd544f57bd6d1467d30e954d P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:44.519924 9678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af5c5a6cae4e4b6ea3eece2e21a587ec" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:44.526378 9539 raft_consensus.cc:3060] T af5c5a6cae4e4b6ea3eece2e21a587ec P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:44.526568 9678 raft_consensus.cc:3060] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:44.497217 9620 leader_election.cc:304] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:44.528617 11234 raft_consensus.cc:2749] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:44.533484 9678 raft_consensus.cc:2468] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:44.536371 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9e66dacf5794b6195575556be889bce" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:44.538877 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f337e9698fe64282a684e90d9c61da5a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:44.544011 9539 raft_consensus.cc:2468] T af5c5a6cae4e4b6ea3eece2e21a587ec P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:44.545110 9767 leader_election.cc:304] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:44.546813 11255 raft_consensus.cc:2804] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:44.547528 11255 raft_consensus.cc:697] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:44.544150 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_18f9a8030cea4cc0abedb81cd5accc59 [id=5c4f226dc33b428e92ca77959e075d7c]
I20260504 13:56:44.557438 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_2099388b6e2a455295be76a6d072758e [id=63e34a66cd644ab7bede27373e321714]
I20260504 13:56:44.559325 10013 tablet_bootstrap.cc:492] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:44.559760 10013 ts_tablet_manager.cc:1403] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.142s user 0.009s sys 0.002s
I20260504 13:56:44.562045 10013 raft_consensus.cc:359] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:44.563467 10013 raft_consensus.cc:385] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:44.563753 10013 raft_consensus.cc:740] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:44.565698 10013 consensus_queue.cc:260] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:44.548476 11255 consensus_queue.cc:237] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:44.567785 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_9bc232a8e4dd4df1a36b35d09dd50d0a [id=66faab34fb324213a874ea6fae3c96a1]
I20260504 13:56:44.568863 11432 raft_consensus.cc:493] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.569233 11432 raft_consensus.cc:515] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:44.572578 11432 leader_election.cc:290] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:44.575522 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ddf890c17964e7f90409ebe3c4034ef" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:44.584568 10013 ts_tablet_manager.cc:1434] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.025s user 0.004s sys 0.001s
I20260504 13:56:44.586269 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_0134ffe7086440ccbcadd076d8fe5d01 [id=693e9b6ed89b4ab7b9d5dc7402e17f20]
I20260504 13:56:44.587369 9931 tablet_bootstrap.cc:492] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:44.587779 9931 ts_tablet_manager.cc:1403] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.288s user 0.010s sys 0.000s
I20260504 13:56:44.588276 10013 tablet_bootstrap.cc:492] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:44.590085 9931 raft_consensus.cc:359] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.591040 9931 raft_consensus.cc:385] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:44.591291 9675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ddf890c17964e7f90409ebe3c4034ef" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:44.598186 9834 raft_consensus.cc:3060] T 7a89bac060214d30924398007dfa1fa1 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:44.605911 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568]
I20260504 13:56:44.611830 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_f00a5967b1b7422b830dcf11db2acbd8 [id=6a34d9af17c84984b479bbd10025fba5]
I20260504 13:56:44.613232 11234 raft_consensus.cc:493] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.613921 11234 raft_consensus.cc:515] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.615957 11234 leader_election.cc:290] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:44.621562 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_5636294717ff4c63939c2cf9facfc788 [id=6acf451855024161b7826094f8387a9c]
W20260504 13:56:44.613402 9620 leader_election.cc:343] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.625905 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_93e03451ea09440d847c57c026230518 [id=6ad5facbe9ca40eeb7746baef5c566a7]
I20260504 13:56:44.625088 9620 leader_election.cc:304] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:44.627115 11234 raft_consensus.cc:2749] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:44.628391 9620 leader_election.cc:343] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.631124 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_8ccb9fc27b374611b20008514d3ad483 [id=6c295b6b72c94ca0b5f1805a555e7f77]
I20260504 13:56:44.630580 9620 leader_election.cc:304] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:44.635494 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_780e641042db4fd5aeb8af8b1aa08c72 [id=6cb26719d5fe45ccbdef18571c902228]
I20260504 13:56:44.636224 11234 raft_consensus.cc:2749] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:44.638125 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_6462cb42c34a4eb8976d8d2a4f82d434 [id=6e3151859b6d4da3a477367687862671]
I20260504 13:56:44.640714 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_49715b69b840473b9f7b0fdabcc0084e [id=700b914cd7ec44e791cd0a27d4b9bf81]
I20260504 13:56:44.653791 10013 tablet_bootstrap.cc:654] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:44.659811 11532 raft_consensus.cc:493] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.660267 11532 raft_consensus.cc:515] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.662236 11532 leader_election.cc:290] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:44.591306 9931 raft_consensus.cc:740] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:44.672199 9931 consensus_queue.cc:260] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.673658 9931 ts_tablet_manager.cc:1434] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.086s user 0.004s sys 0.000s
I20260504 13:56:44.674489 9931 tablet_bootstrap.cc:492] T a2bfb681e6824e9f9bdcbea23061d57a P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:44.675871 9473 leader_election.cc:304] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:44.676939 11539 raft_consensus.cc:2749] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:44.679163 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_11d8ab61a6044154bb599875a0bc9004 [id=73e12524eb9e416782bf9f7fb58c93bd]
I20260504 13:56:44.683094 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d6db45d2e5c41a3b30cede3301694d3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:44.684649 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b942037c0e45a6b3cccba288b90b00" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:44.685616 9618 leader_election.cc:343] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:44.686168 9618 leader_election.cc:343] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.689038 9588 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:44.683326 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_7dbd11258f34498e82a86f7e3c72f470 [id=7bf52f9bd2a042e59fc4ff612d9846e9]
I20260504 13:56:44.689587 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b942037c0e45a6b3cccba288b90b00" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:44.690701 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d6db45d2e5c41a3b30cede3301694d3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:44.691349 11434 consensus_queue.cc:1048] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:44.692812 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_df8874a20e834872a12300a4f5628273 [id=7fc0413d79184a2dbe5c470b8506b4e1]
I20260504 13:56:44.696974 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_4842f50414d34bdb8a10802bc059b421 [id=7fc23620ef7342eea52bcce264eb12b3]
I20260504 13:56:44.699026 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_24dab2dd1f45402786d80cba80bbad82 [id=80ca58df22214450b894afea377e9c16]
I20260504 13:56:44.704480 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_cdee25533fe54da0a74b421714b0d7c4 [id=81df524d85774544807ba1902be1aae1]
I20260504 13:56:44.705991 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]
I20260504 13:56:44.713440 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_1c9b2e63142440a5a45018d187a6cded [id=8a427e40b3374dafa140c9156edcc660]
I20260504 13:56:44.721421 9184 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 88619eaa917d4c33a910fc1347ab126e (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
W20260504 13:56:44.696110 9620 leader_election.cc:343] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.725812 9620 leader_election.cc:304] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:44.748996 11532 raft_consensus.cc:2749] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:44.762844 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 66585us
I20260504 13:56:44.773536 10013 tablet_bootstrap.cc:492] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:44.773933 10013 ts_tablet_manager.cc:1403] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.186s user 0.013s sys 0.003s
I20260504 13:56:44.776253 10013 raft_consensus.cc:359] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.778080 10013 raft_consensus.cc:385] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:44.778335 10013 raft_consensus.cc:740] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
W20260504 13:56:44.778620 9620 leader_election.cc:343] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.779084 9620 leader_election.cc:304] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:44.778895 10013 consensus_queue.cc:260] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.779764 11532 raft_consensus.cc:2749] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:44.780815 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_9e9ef3353dcf4b1fadaa2f51bc8f7041 [id=8dcabd7d67334598926f927a62e630d6]
I20260504 13:56:44.781119 11532 raft_consensus.cc:493] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.781538 11532 raft_consensus.cc:515] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.783245 11532 leader_election.cc:290] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:44.783949 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ff0d52a90c4056880190dbd936cb8e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:44.785348 9618 leader_election.cc:343] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:44.787890 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_b59e5e285d50401e94d51afeb4cbc500 [id=8f16af9aec63405cbae2a91b84fa3009]
I20260504 13:56:44.798580 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_a992fb6e038e4408a781d6d8ead37f54 [id=937a02c37dbd45d5bc92979ceaee1c06]
I20260504 13:56:44.804759 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]
I20260504 13:56:44.806007 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_443402c40ae5443daf4ba4258b847303 [id=98126e5c27cd4d3da57fcd88d481de79]
I20260504 13:56:44.806768 9184 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4]). Expected version 1 got 0
I20260504 13:56:44.810295 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_10e92469b9cb4e9b965d0a10cfeecca8 [id=99da5017cead4ddeaa3441064b1ed716]
I20260504 13:56:44.822150 10013 ts_tablet_manager.cc:1434] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20260504 13:56:44.825418 10013 tablet_bootstrap.cc:492] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:44.832168 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_655e2e8cf3544fbeb0ce33829bb2c8a0 [id=9db1e031520d405ea4989e30d5194911]
I20260504 13:56:44.833446 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4]
I20260504 13:56:44.848102 10013 tablet_bootstrap.cc:654] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:44.835263 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_225ad80b8318432b88a729064c88e8ec [id=a3cca003025745cebf8e7d403d0778cf]
I20260504 13:56:44.852942 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_e55147137c794be4bff21afce5d36bbc [id=b45733fe3c5547bea00e3967be81ca35]
I20260504 13:56:44.860169 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_6e41cb79b95848ba8c7b1f05f961c7f3 [id=b91fe7c5c3844c109756842f93098699]
I20260504 13:56:44.861322 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]
I20260504 13:56:44.862556 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_862ae682dfb44737a72e7c5df628a957 [id=bbef146162cb4c30a3ba42ca2a36fe9f]
I20260504 13:56:44.863715 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_5bc32a610edd49ab9641c3726f53f5c6 [id=be5c761518674e7782bfff73366b1675]
I20260504 13:56:44.881409 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_e116f9abbb23419d9a3365f7d0e17db9 [id=bee8acaa102249578514a3d4e5f876d3]
I20260504 13:56:44.890709 11434 consensus_queue.cc:1048] T 92480654807d495b879576b0195871bd P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:44.896405 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ff0d52a90c4056880190dbd936cb8e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:44.896814 10013 tablet_bootstrap.cc:492] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:44.897320 10013 ts_tablet_manager.cc:1403] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.072s user 0.013s sys 0.000s
I20260504 13:56:44.900422 10013 raft_consensus.cc:359] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.901062 10013 raft_consensus.cc:385] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:44.901384 10013 raft_consensus.cc:740] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:44.902072 10013 consensus_queue.cc:260] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:44.903461 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:44.904510 10013 ts_tablet_manager.cc:1434] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260504 13:56:44.907019 9931 tablet_bootstrap.cc:654] T a2bfb681e6824e9f9bdcbea23061d57a P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:44.910681 10013 tablet_bootstrap.cc:492] T 36c8e9ba29a1466082a3e8bedec87b91 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:44.915680 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]
I20260504 13:56:44.915973 9184 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
I20260504 13:56:44.920387 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_849bbe7902ca44969c5208828774fa06 [id=c146e84e0e844deda9af40562f980047]
I20260504 13:56:44.935968 11532 raft_consensus.cc:493] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:44.936443 11532 raft_consensus.cc:515] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:44.938189 11532 leader_election.cc:290] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:44.939812 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_861ff05bb9e144ea85a3fb7d75f68749 [id=c4b5a9431b8b4d33b41fa27a357e1f46]
I20260504 13:56:44.940142 11532 consensus_queue.cc:1048] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:44.942289 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_363d6634e42e4dd6a5686c1f0e401ef2 [id=d0d8b54fb69540589420b87f90b4c823]
I20260504 13:56:44.949057 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf11f17e95dc411294c755e17c57811e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:44.952214 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_3387191fcd6046abbf257447bfbb27f7 [id=d195f5bef6ce49f1b87a4bf5f516eebb]
I20260504 13:56:44.957198 10013 tablet_bootstrap.cc:654] T 36c8e9ba29a1466082a3e8bedec87b91 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:44.963297 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 72911.8us
I20260504 13:56:44.980198 9184 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38]). Expected version 1 got 0
I20260504 13:56:45.001497 10013 tablet_bootstrap.cc:492] T 36c8e9ba29a1466082a3e8bedec87b91 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:45.001959 10013 ts_tablet_manager.cc:1403] T 36c8e9ba29a1466082a3e8bedec87b91 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.091s user 0.013s sys 0.002s
I20260504 13:56:45.004245 10013 raft_consensus.cc:359] T 36c8e9ba29a1466082a3e8bedec87b91 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.005752 10013 raft_consensus.cc:385] T 36c8e9ba29a1466082a3e8bedec87b91 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:45.007692 10013 raft_consensus.cc:740] T 36c8e9ba29a1466082a3e8bedec87b91 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
W20260504 13:56:45.008314 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 57784.9us
I20260504 13:56:45.010262 9531 raft_consensus.cc:3060] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:45.025364 9931 tablet_bootstrap.cc:492] T a2bfb681e6824e9f9bdcbea23061d57a P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:45.025790 9931 ts_tablet_manager.cc:1403] T a2bfb681e6824e9f9bdcbea23061d57a P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.351s user 0.011s sys 0.000s
I20260504 13:56:45.009279 10013 consensus_queue.cc:260] T 36c8e9ba29a1466082a3e8bedec87b91 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.032702 10013 ts_tablet_manager.cc:1434] T 36c8e9ba29a1466082a3e8bedec87b91 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.030s user 0.004s sys 0.000s
I20260504 13:56:45.037281 9931 raft_consensus.cc:359] T a2bfb681e6824e9f9bdcbea23061d57a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.037889 9931 raft_consensus.cc:385] T a2bfb681e6824e9f9bdcbea23061d57a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:45.038137 9931 raft_consensus.cc:740] T a2bfb681e6824e9f9bdcbea23061d57a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:45.038769 9931 consensus_queue.cc:260] T a2bfb681e6824e9f9bdcbea23061d57a P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.039274 10013 tablet_bootstrap.cc:492] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:45.040815 9931 ts_tablet_manager.cc:1434] T a2bfb681e6824e9f9bdcbea23061d57a P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260504 13:56:45.043589 9184 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 3 got 0
I20260504 13:56:45.046690 9184 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet de6714ee86d64112baf13dd1b2b87582 (table default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469]). Expected version 1 got 0
I20260504 13:56:45.073488 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469]
I20260504 13:56:45.071507 10013 tablet_bootstrap.cc:654] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:45.075693 9931 tablet_bootstrap.cc:492] T b84c3bedc7a84e0d8a873fb15689b314 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:45.077934 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_6b0124eccb02454b9195b8ddaae5bd38 [id=d47a2c6a9cb34ba4ade10fc23913bf7d]
I20260504 13:56:45.080245 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_d6a5a22d78264e799e80d31d5749600a [id=d63acf266ca440729128a717eed32433]
I20260504 13:56:45.081423 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_91cc6c38c9ff43dca25814664068b1c6 [id=da5c427e75e74ea1ae3a5961ed27e0de]
I20260504 13:56:45.081472 9931 tablet_bootstrap.cc:654] T b84c3bedc7a84e0d8a873fb15689b314 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:45.096271 9931 tablet_bootstrap.cc:492] T b84c3bedc7a84e0d8a873fb15689b314 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:45.096773 9931 ts_tablet_manager.cc:1403] T b84c3bedc7a84e0d8a873fb15689b314 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.000s
I20260504 13:56:45.098997 9931 raft_consensus.cc:359] T b84c3bedc7a84e0d8a873fb15689b314 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.099741 9931 raft_consensus.cc:385] T b84c3bedc7a84e0d8a873fb15689b314 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:45.099989 9931 raft_consensus.cc:740] T b84c3bedc7a84e0d8a873fb15689b314 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:45.100513 9931 consensus_queue.cc:260] T b84c3bedc7a84e0d8a873fb15689b314 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.101833 9931 ts_tablet_manager.cc:1434] T b84c3bedc7a84e0d8a873fb15689b314 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260504 13:56:45.138098 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_8cae213b77994b9fba63b8a7c5f27249 [id=df86bccb4ca8414c90f9fd8753724d89]
I20260504 13:56:45.139631 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1]
I20260504 13:56:45.150239 10013 tablet_bootstrap.cc:492] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:45.150633 10013 ts_tablet_manager.cc:1403] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.112s user 0.015s sys 0.000s
I20260504 13:56:45.153272 11548 raft_consensus.cc:493] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:45.153224 10013 raft_consensus.cc:359] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.153518 9184 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 2ddf890c17964e7f90409ebe3c4034ef (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]). Expected version 2 got 0
I20260504 13:56:45.153872 10013 raft_consensus.cc:385] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:45.154145 10013 raft_consensus.cc:740] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:45.154248 9184 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]). Expected version 1 got 0
I20260504 13:56:45.153919 11548 raft_consensus.cc:515] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.154793 10013 consensus_queue.cc:260] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.155849 9184 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet e53381ee0e7f4e9781b833b435bca831 (table default.Table_9b40ba316993413181a9743fb7bab7fb [id=fb4e5ce5526e44de8effdd766f4c257f]). Expected version 1 got 0
I20260504 13:56:45.156298 11548 leader_election.cc:290] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:45.157655 10013 ts_tablet_manager.cc:1434] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260504 13:56:45.157647 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da61c91e10264fa49fe397d9a9b014bb" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:45.158314 9834 raft_consensus.cc:2393] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 1: Already voted for candidate 5beceb6317444247b31b44a6581707c6 in this term.
I20260504 13:56:45.164438 9931 tablet_bootstrap.cc:492] T 5cb30835150949f39f6e7e7563381763 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:45.169812 9931 tablet_bootstrap.cc:654] T 5cb30835150949f39f6e7e7563381763 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:45.186894 10013 tablet_bootstrap.cc:492] T f71f444ce9214e3f98eff6ac8ca7dde7 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:45.187321 9931 tablet_bootstrap.cc:492] T 5cb30835150949f39f6e7e7563381763 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:45.187690 9931 ts_tablet_manager.cc:1403] T 5cb30835150949f39f6e7e7563381763 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.000s
I20260504 13:56:45.188364 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_06ecf48a04d94ac8a7e954ff2c260dbf [id=e11be55403d24c8bb35f4708da2e9a78]
I20260504 13:56:45.189670 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_999e37d0dab54ca68c0fa66f74582109 [id=e37fd7e36e604a60832acf303f0ecb7f]
I20260504 13:56:45.189811 9931 raft_consensus.cc:359] T 5cb30835150949f39f6e7e7563381763 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.190378 9931 raft_consensus.cc:385] T 5cb30835150949f39f6e7e7563381763 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:45.190613 9931 raft_consensus.cc:740] T 5cb30835150949f39f6e7e7563381763 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:45.191313 9931 consensus_queue.cc:260] T 5cb30835150949f39f6e7e7563381763 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.192646 9931 ts_tablet_manager.cc:1434] T 5cb30835150949f39f6e7e7563381763 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260504 13:56:45.193449 9931 tablet_bootstrap.cc:492] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:45.193534 10013 tablet_bootstrap.cc:654] T f71f444ce9214e3f98eff6ac8ca7dde7 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:45.205574 9931 tablet_bootstrap.cc:654] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:45.227615 9931 tablet_bootstrap.cc:492] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:45.228058 9931 ts_tablet_manager.cc:1403] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.003s
I20260504 13:56:45.230244 9931 raft_consensus.cc:359] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:45.230907 9931 raft_consensus.cc:385] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:45.231158 9931 raft_consensus.cc:740] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:45.231937 9931 consensus_queue.cc:260] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:45.233726 9931 ts_tablet_manager.cc:1434] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260504 13:56:45.234541 9931 tablet_bootstrap.cc:492] T c1a01231bb044433ae5827d07a6592f8 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:45.191102 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_89c38f9ba46e48fab397a7f3d5d1b805 [id=e4f6621892874f979d1c579d487cba44]
I20260504 13:56:45.246815 9931 tablet_bootstrap.cc:654] T c1a01231bb044433ae5827d07a6592f8 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:45.262171 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]
I20260504 13:56:45.263744 10013 tablet_bootstrap.cc:492] T f71f444ce9214e3f98eff6ac8ca7dde7 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:45.264107 10013 ts_tablet_manager.cc:1403] T f71f444ce9214e3f98eff6ac8ca7dde7 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.077s user 0.008s sys 0.000s
I20260504 13:56:45.281528 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_737cc12f07144a6c92dc2974693a649d [id=e9103e8b725b4e81b6cf094395beef3b]
I20260504 13:56:45.298597 9931 tablet_bootstrap.cc:492] T c1a01231bb044433ae5827d07a6592f8 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:45.299015 9931 ts_tablet_manager.cc:1403] T c1a01231bb044433ae5827d07a6592f8 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.003s
W20260504 13:56:44.944837 9620 leader_election.cc:343] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:45.304487 9620 leader_election.cc:304] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:45.305106 9931 raft_consensus.cc:359] T c1a01231bb044433ae5827d07a6592f8 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.305756 9931 raft_consensus.cc:385] T c1a01231bb044433ae5827d07a6592f8 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:45.306005 9931 raft_consensus.cc:740] T c1a01231bb044433ae5827d07a6592f8 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
W20260504 13:56:45.305917 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 361384us
I20260504 13:56:45.306600 9931 consensus_queue.cc:260] T c1a01231bb044433ae5827d07a6592f8 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:44.950333 9618 leader_election.cc:343] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:45.311944 9618 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 361503us
I20260504 13:56:45.324714 9881 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:45.324772 9834 raft_consensus.cc:3060] T 5cb30835150949f39f6e7e7563381763 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:45.334560 9539 raft_consensus.cc:3060] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:45.339685 9931 ts_tablet_manager.cc:1434] T c1a01231bb044433ae5827d07a6592f8 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20260504 13:56:45.344900 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf11f17e95dc411294c755e17c57811e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:45.357021 11560 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 806407748d564439807fe5d73db9abda)
I20260504 13:56:45.357949 11560 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:45.360415 11560 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:45.362159 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 5 candidate_status { last_received { term: 4 index: 368 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:45.363312 11561 raft_consensus.cc:2749] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:45.387997 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:45.399677 11566 raft_consensus.cc:493] T b84c3bedc7a84e0d8a873fb15689b314 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:45.400264 11566 raft_consensus.cc:515] T b84c3bedc7a84e0d8a873fb15689b314 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.404800 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_a5fa0a0bead94aa3b02d6175fefde171 [id=e9fbd43d30ae4d1dbeea534025125b91]
I20260504 13:56:45.406119 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_1e6b4c9e684e45b3b2510d3979c1b5c4 [id=ecfa480ab0ea40008ff4507ed6c5b964]
I20260504 13:56:45.159039 9184 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]). Expected version 1 got 0
I20260504 13:56:45.428378 9531 raft_consensus.cc:3060] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:45.440496 11566 leader_election.cc:290] T b84c3bedc7a84e0d8a873fb15689b314 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:45.444607 11476 raft_consensus.cc:493] T c1a01231bb044433ae5827d07a6592f8 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:45.445129 11476 raft_consensus.cc:515] T c1a01231bb044433ae5827d07a6592f8 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.448500 11476 leader_election.cc:290] T c1a01231bb044433ae5827d07a6592f8 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:45.406909 11532 raft_consensus.cc:493] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:45.452391 9675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1a01231bb044433ae5827d07a6592f8" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:45.459504 9675 raft_consensus.cc:2393] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:45.459227 11532 raft_consensus.cc:515] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:45.450889 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b84c3bedc7a84e0d8a873fb15689b314" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:45.461514 11532 leader_election.cc:290] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:45.463966 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b84c3bedc7a84e0d8a873fb15689b314" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:45.464845 9532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1a01231bb044433ae5827d07a6592f8" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:45.465751 10013 raft_consensus.cc:359] T f71f444ce9214e3f98eff6ac8ca7dde7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.466462 10013 raft_consensus.cc:385] T f71f444ce9214e3f98eff6ac8ca7dde7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:45.466537 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6323e1c10e7643f8a28f04e285f620e9" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:45.466796 10013 raft_consensus.cc:740] T f71f444ce9214e3f98eff6ac8ca7dde7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:45.467537 10013 consensus_queue.cc:260] T f71f444ce9214e3f98eff6ac8ca7dde7 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:45.468376 9618 leader_election.cc:343] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:45.498243 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_2dc0381bd6c84ab59d5bb422de89c0e2 [id=ed756f69850642188b881bd3c680a622]
I20260504 13:56:45.499595 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_09fd612a3c4a4303b8f05530e7d49819 [id=eda14d7d5f7d4391a8c470677ffab4ea]
W20260504 13:56:45.504171 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 47189.1us
I20260504 13:56:45.510438 11476 raft_consensus.cc:493] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:45.510951 11476 raft_consensus.cc:515] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:45.513991 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a862a43c5ac443d9bac98e0472ca23b2" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:45.528749 9829 raft_consensus.cc:3060] T c1a01231bb044433ae5827d07a6592f8 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:45.534758 9931 tablet_bootstrap.cc:492] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:45.535681 9828 raft_consensus.cc:3060] T b84c3bedc7a84e0d8a873fb15689b314 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:45.539822 11476 leader_election.cc:290] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:45.540211 9931 tablet_bootstrap.cc:654] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:45.540547 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_72e3a254561b4cae927e6b6bcacd87b6 [id=f227576f105c4b1fbd459b3638ce684b]
I20260504 13:56:45.542227 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_7f912e1d4381428e9572cbb4b13f4eec [id=fb18c55d21704b2ab04dbf35d21377bf]
I20260504 13:56:45.543467 11568 raft_consensus.cc:493] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:45.543918 11568 raft_consensus.cc:515] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.544399 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_9b40ba316993413181a9743fb7bab7fb [id=fb4e5ce5526e44de8effdd766f4c257f]
I20260504 13:56:45.582968 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da61c91e10264fa49fe397d9a9b014bb" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:45.583837 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_7db59fa280b6442b99cf24677b57df83 [id=fdb915ebe83443fd8d2f7f287385f6a4]
I20260504 13:56:45.604935 11568 leader_election.cc:290] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:45.606729 9768 leader_election.cc:343] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 21d6358b451d4b8d87a22e30e0780f44
I20260504 13:56:45.506448 9767 leader_election.cc:304] T b84c3bedc7a84e0d8a873fb15689b314 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:45.594143 11567 raft_consensus.cc:493] T a2bfb681e6824e9f9bdcbea23061d57a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:45.613622 11567 raft_consensus.cc:515] T a2bfb681e6824e9f9bdcbea23061d57a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.615485 11567 leader_election.cc:290] T a2bfb681e6824e9f9bdcbea23061d57a P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:45.617475 11567 raft_consensus.cc:2749] T b84c3bedc7a84e0d8a873fb15689b314 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:45.597034 11480 consensus_queue.cc:1048] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:45.585891 10998 catalog_manager.cc:679] Loaded metadata for table default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]
I20260504 13:56:45.620939 11568 consensus_queue.cc:1048] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:45.624333 9736 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
W20260504 13:56:45.627185 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 120927us
I20260504 13:56:45.627832 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2bfb681e6824e9f9bdcbea23061d57a" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:45.629227 9767 leader_election.cc:304] T c1a01231bb044433ae5827d07a6592f8 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:45.629899 11567 raft_consensus.cc:2749] T c1a01231bb044433ae5827d07a6592f8 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:45.632180 10013 ts_tablet_manager.cc:1434] T f71f444ce9214e3f98eff6ac8ca7dde7 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.368s user 0.006s sys 0.000s
I20260504 13:56:45.658762 9931 tablet_bootstrap.cc:492] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:45.659251 9931 ts_tablet_manager.cc:1403] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.125s user 0.017s sys 0.012s
I20260504 13:56:45.661650 9931 raft_consensus.cc:359] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.662310 9931 raft_consensus.cc:385] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:45.662595 9931 raft_consensus.cc:740] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:45.663272 9931 consensus_queue.cc:260] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.688439 10013 tablet_bootstrap.cc:492] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:45.699640 11579 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:45.703715 11579 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
W20260504 13:56:45.710345 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 81177.3us
I20260504 13:56:45.713176 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:45.720396 11579 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:56:45.721777 10810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:45.724135 9328 leader_election.cc:304] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec9f4c3a43d84fc9a420dd10723c01a1; no voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda
I20260504 13:56:45.724912 11579 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 13:56:45.757941 11480 raft_consensus.cc:493] T 36c8e9ba29a1466082a3e8bedec87b91 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:45.758378 11480 raft_consensus.cc:515] T 36c8e9ba29a1466082a3e8bedec87b91 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:45.760371 11480 leader_election.cc:290] T 36c8e9ba29a1466082a3e8bedec87b91 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:45.789497 9931 ts_tablet_manager.cc:1434] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.130s user 0.004s sys 0.000s
I20260504 13:56:45.790349 9931 tablet_bootstrap.cc:492] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:45.809082 10998 tablet_loader.cc:96] loaded metadata for tablet 03e1d998c4314db3b3682a0510393749 (table default.Table_f00a5967b1b7422b830dcf11db2acbd8 [id=6a34d9af17c84984b479bbd10025fba5])
I20260504 13:56:45.810242 10998 tablet_loader.cc:96] loaded metadata for tablet 03fe4ce4f52f48b7b7a804c269e0fd4e (table default.Table_963c93d5d8ed49e6a2052fa1e652da91 [id=399a65bb2fa6428296341de74e964518])
I20260504 13:56:45.823557 9531 raft_consensus.cc:3060] T 36c8e9ba29a1466082a3e8bedec87b91 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:45.823989 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a862a43c5ac443d9bac98e0472ca23b2" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:45.837253 10013 tablet_bootstrap.cc:654] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:45.854911 10998 tablet_loader.cc:96] loaded metadata for tablet 0be795f04dd54cf19869c937b91d0ddf (table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f])
I20260504 13:56:45.857539 10998 tablet_loader.cc:96] loaded metadata for tablet 184fefa651fc411fb5107cc923c02535 (table default.Table_f8a23a8252f84cacb13b338f9d9dc9fc [id=39be2c4d16aa41c0832bfeb23269243a])
I20260504 13:56:45.859359 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36c8e9ba29a1466082a3e8bedec87b91" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:45.841393 9588 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:45.844525 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21d6358b451d4b8d87a22e30e0780f44" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:45.845238 9532 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2bfb681e6824e9f9bdcbea23061d57a" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:45.864894 9532 raft_consensus.cc:2393] T a2bfb681e6824e9f9bdcbea23061d57a P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 5beceb6317444247b31b44a6581707c6 in this term.
I20260504 13:56:45.864252 9539 raft_consensus.cc:2468] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
W20260504 13:56:45.865928 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:45.904505 10998 tablet_loader.cc:96] loaded metadata for tablet 1c12ffa802b840b1937f6f796083c478 (table default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1])
I20260504 13:56:45.906478 10998 tablet_loader.cc:96] loaded metadata for tablet 1fbe09b4078c4eebb52d7d00bc87371c (table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee])
I20260504 13:56:45.927923 9826 raft_consensus.cc:3060] T a2bfb681e6824e9f9bdcbea23061d57a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:45.930627 9931 tablet_bootstrap.cc:654] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:45.933844 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6323e1c10e7643f8a28f04e285f620e9" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:45.943003 9767 leader_election.cc:304] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:45.943753 11567 raft_consensus.cc:2749] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:45.956404 10998 tablet_loader.cc:96] loaded metadata for tablet 23ff0d52a90c4056880190dbd936cb8e (table default.Table_24398dd8ee63413ebb585587353ea155 [id=1d2bbdaf6382433cb8c2a55f62f0388d])
I20260504 13:56:45.958190 10998 tablet_loader.cc:96] loaded metadata for tablet 274bb204041c4e1085395c7bd938aaf1 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5])
I20260504 13:56:45.980191 11567 consensus_queue.cc:1048] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:45.994289 9473 leader_election.cc:304] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:45.995609 11586 raft_consensus.cc:2749] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:45.996876 11586 consensus_queue.cc:1048] T c1a01231bb044433ae5827d07a6592f8 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:46.013091 10013 tablet_bootstrap.cc:492] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:46.013576 10013 ts_tablet_manager.cc:1403] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.325s user 0.014s sys 0.007s
I20260504 13:56:46.015321 10998 tablet_loader.cc:96] loaded metadata for tablet 2a629c2a69084b6b98984c8d374eeb1e (table default.Table_a5fa0a0bead94aa3b02d6175fefde171 [id=e9fbd43d30ae4d1dbeea534025125b91])
I20260504 13:56:46.017382 10998 tablet_loader.cc:96] loaded metadata for tablet 2c09e9076214405eaa8846fe08158bb3 (table default.Table_6e41cb79b95848ba8c7b1f05f961c7f3 [id=b91fe7c5c3844c109756842f93098699])
I20260504 13:56:46.018548 10998 tablet_loader.cc:96] loaded metadata for tablet 2ddf890c17964e7f90409ebe3c4034ef (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e])
I20260504 13:56:46.018735 10013 raft_consensus.cc:359] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.019552 10013 raft_consensus.cc:385] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:46.019846 10013 raft_consensus.cc:740] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:46.020632 10013 consensus_queue.cc:260] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.039870 10013 ts_tablet_manager.cc:1434] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260504 13:56:46.047981 10013 tablet_bootstrap.cc:492] T 57983fcecdb64a42884f797b4db57df0 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:46.047959 10998 tablet_loader.cc:96] loaded metadata for tablet 2f6b3e969760442283c3191e062c1839 (table default.Table_5bc32a610edd49ab9641c3726f53f5c6 [id=be5c761518674e7782bfff73366b1675])
I20260504 13:56:46.059774 10013 tablet_bootstrap.cc:654] T 57983fcecdb64a42884f797b4db57df0 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:46.061787 10998 tablet_loader.cc:96] loaded metadata for tablet 3404ba04d3ec4a44a5e625536b941b58 (table default.Table_7db59fa280b6442b99cf24677b57df83 [id=fdb915ebe83443fd8d2f7f287385f6a4])
I20260504 13:56:46.064404 10998 tablet_loader.cc:96] loaded metadata for tablet 36c8e9ba29a1466082a3e8bedec87b91 (table default.Table_1665799e268a496d9ffc5568fa34a763 [id=403cbbc8a1304e5695a89d3697932c26])
W20260504 13:56:46.091658 10690 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.238.252:44781 timed out after 0.727s (SENT) [suppressed 1 similar messages]
I20260504 13:56:46.115386 10998 tablet_loader.cc:96] loaded metadata for tablet 37b1664312d347288680d97b62c2aecb (table default.Table_861ff05bb9e144ea85a3fb7d75f68749 [id=c4b5a9431b8b4d33b41fa27a357e1f46])
I20260504 13:56:46.118889 10998 tablet_loader.cc:96] loaded metadata for tablet 3d9bd2722d23420ea0eac7664357a60c (table default.Table_821e9f466c974a2fbf24c07f8c266e62 [id=469bd5cb4e7f4a8a9c7af947a834ca7b])
W20260504 13:56:46.133366 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 137360us
I20260504 13:56:46.134867 10013 tablet_bootstrap.cc:492] T 57983fcecdb64a42884f797b4db57df0 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:46.135298 10013 ts_tablet_manager.cc:1403] T 57983fcecdb64a42884f797b4db57df0 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.087s user 0.011s sys 0.004s
W20260504 13:56:46.140949 9473 consensus_peers.cc:597] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e4e583afc804212bece2f727411cc14. This is attempt 6: this message will repeat every 5th retry.
I20260504 13:56:46.140262 10013 raft_consensus.cc:359] T 57983fcecdb64a42884f797b4db57df0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.142378 10013 raft_consensus.cc:385] T 57983fcecdb64a42884f797b4db57df0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:46.142630 10013 raft_consensus.cc:740] T 57983fcecdb64a42884f797b4db57df0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:46.143271 10013 consensus_queue.cc:260] T 57983fcecdb64a42884f797b4db57df0 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.145306 10013 ts_tablet_manager.cc:1434] T 57983fcecdb64a42884f797b4db57df0 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260504 13:56:46.146396 10013 tablet_bootstrap.cc:492] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:46.157536 11575 raft_consensus.cc:493] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:46.157953 11575 raft_consensus.cc:515] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.159761 11575 leader_election.cc:290] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:46.160678 10998 tablet_loader.cc:96] loaded metadata for tablet 4a1a77ad56634e068401a2bfbc977761 (table default.Table_9bc232a8e4dd4df1a36b35d09dd50d0a [id=66faab34fb324213a874ea6fae3c96a1])
I20260504 13:56:46.161442 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "644170f17f1b45a19677d958b9745e16" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:46.161792 10998 tablet_loader.cc:96] loaded metadata for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38])
I20260504 13:56:46.162843 10998 tablet_loader.cc:96] loaded metadata for tablet 4cf1504f0202481ba8b08f9bec5bc3d9 (table default.Table_737cc12f07144a6c92dc2974693a649d [id=e9103e8b725b4e81b6cf094395beef3b])
W20260504 13:56:46.162941 9618 leader_election.cc:343] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:46.169548 9931 tablet_bootstrap.cc:492] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:46.169945 9931 ts_tablet_manager.cc:1403] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.380s user 0.013s sys 0.004s
W20260504 13:56:46.170393 9620 leader_election.cc:343] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:46.170850 9620 leader_election.cc:304] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:46.172382 9931 raft_consensus.cc:359] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.172976 9931 raft_consensus.cc:385] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:46.173224 9931 raft_consensus.cc:740] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:46.173226 11564 raft_consensus.cc:2749] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:46.173763 9931 consensus_queue.cc:260] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.175796 9931 ts_tablet_manager.cc:1434] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260504 13:56:46.178067 9767 leader_election.cc:304] T a2bfb681e6824e9f9bdcbea23061d57a P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:46.179039 9767 leader_election.cc:304] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:46.179988 11255 raft_consensus.cc:2749] T a2bfb681e6824e9f9bdcbea23061d57a P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:46.180500 11255 raft_consensus.cc:2804] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:46.180809 11255 raft_consensus.cc:493] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:46.181109 11255 raft_consensus.cc:3060] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:46.193959 9931 tablet_bootstrap.cc:492] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:46.198565 9830 raft_consensus.cc:3060] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:46.201078 9620 leader_election.cc:343] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:46.202229 9620 leader_election.cc:304] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:46.205258 11564 raft_consensus.cc:2749] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:46.211370 11564 consensus_queue.cc:1048] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:46.212968 9533 raft_consensus.cc:3060] T f71f444ce9214e3f98eff6ac8ca7dde7 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:46.231778 9931 tablet_bootstrap.cc:654] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:46.254874 10998 tablet_loader.cc:96] loaded metadata for tablet 4fae9274bb8240798f85b79504a227b3 (table default.Table_e116f9abbb23419d9a3365f7d0e17db9 [id=bee8acaa102249578514a3d4e5f876d3])
W20260504 13:56:46.267457 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:46.288031 9827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "644170f17f1b45a19677d958b9745e16" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:46.316799 10998 tablet_loader.cc:96] loaded metadata for tablet 4fdc09a498be470ab04563c62caf7755 (table default.Table_49715b69b840473b9f7b0fdabcc0084e [id=700b914cd7ec44e791cd0a27d4b9bf81])
I20260504 13:56:46.317828 11564 consensus_queue.cc:1048] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:46.328331 10998 tablet_loader.cc:96] loaded metadata for tablet 5254b1061e2b424487f5d309baaee6b6 (table default.Table_798d2540b8f4489fb11763ccaa36bced [id=049b4fa786fc4dbb8e52931dcaaab90f])
I20260504 13:56:46.318462 11255 raft_consensus.cc:515] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.321856 9931 tablet_bootstrap.cc:492] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:46.331789 9931 ts_tablet_manager.cc:1403] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.138s user 0.019s sys 0.001s
I20260504 13:56:46.334275 9931 raft_consensus.cc:359] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.334990 9931 raft_consensus.cc:385] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:46.335322 9931 raft_consensus.cc:740] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:46.336000 9931 consensus_queue.cc:260] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.340888 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36c8e9ba29a1466082a3e8bedec87b91" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:46.346177 11255 leader_election.cc:290] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:46.348021 10013 tablet_bootstrap.cc:700] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent opening tablet: real 0.113s user 0.003s sys 0.000s
I20260504 13:56:46.348683 11564 raft_consensus.cc:493] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:46.348793 10013 tablet_bootstrap.cc:654] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:46.349130 11564 raft_consensus.cc:515] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:46.350445 9768 leader_election.cc:343] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 21d6358b451d4b8d87a22e30e0780f44
I20260504 13:56:46.350934 11564 leader_election.cc:290] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
W20260504 13:56:46.356202 9173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 74453.5us
I20260504 13:56:46.357236 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e29a7365caf04506b15089f10e6c1677" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:46.359817 9618 leader_election.cc:343] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:46.363850 11432 raft_consensus.cc:493] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:46.364365 11432 raft_consensus.cc:515] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.366642 11432 leader_election.cc:290] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:46.385429 10998 tablet_loader.cc:96] loaded metadata for tablet 538c8c2412104c3e8b19394a1cc38818 (table default.Table_7dbd11258f34498e82a86f7e3c72f470 [id=7bf52f9bd2a042e59fc4ff612d9846e9])
W20260504 13:56:46.385905 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 149636us
I20260504 13:56:46.397500 10998 tablet_loader.cc:96] loaded metadata for tablet 564d0fafb49a4b4bb3bdb240c74c3410 (table default.Table_1c9b2e63142440a5a45018d187a6cded [id=8a427e40b3374dafa140c9156edcc660])
I20260504 13:56:46.409397 10013 tablet_bootstrap.cc:492] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:46.409938 10013 ts_tablet_manager.cc:1403] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.264s user 0.014s sys 0.000s
I20260504 13:56:46.412990 10998 tablet_loader.cc:96] loaded metadata for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537])
I20260504 13:56:46.415248 10013 raft_consensus.cc:359] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.417119 10998 tablet_loader.cc:96] loaded metadata for tablet 59f9115b286d44f0b7eac7c74275cb21 (table default.Table_5944297f8c104978bbc6913ecd8f47a4 [id=3f11131cece44f3ebf3cd0f1e69139ae])
I20260504 13:56:46.430038 10013 raft_consensus.cc:385] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:46.430694 10998 tablet_loader.cc:96] loaded metadata for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4])
I20260504 13:56:46.427256 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21d6358b451d4b8d87a22e30e0780f44" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:46.433214 10013 raft_consensus.cc:740] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:46.433363 9533 raft_consensus.cc:3060] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:46.434576 9931 ts_tablet_manager.cc:1434] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.102s user 0.005s sys 0.005s
I20260504 13:56:46.435416 9931 tablet_bootstrap.cc:492] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:46.439829 9533 raft_consensus.cc:2468] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:46.440596 10998 tablet_loader.cc:96] loaded metadata for tablet 6323e1c10e7643f8a28f04e285f620e9 (table default.Table_999e37d0dab54ca68c0fa66f74582109 [id=e37fd7e36e604a60832acf303f0ecb7f])
I20260504 13:56:46.433987 10013 consensus_queue.cc:260] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.446669 9767 leader_election.cc:304] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6
I20260504 13:56:46.447005 9931 tablet_bootstrap.cc:654] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:46.447849 11568 raft_consensus.cc:2804] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:46.448334 11568 raft_consensus.cc:697] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:46.451498 10013 ts_tablet_manager.cc:1434] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20260504 13:56:46.453195 11564 raft_consensus.cc:493] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:46.454177 11564 raft_consensus.cc:515] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.460276 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddd30e9259a04d59b5c1bb82aabf7952" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:46.461478 11589 raft_consensus.cc:493] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:46.461913 11589 raft_consensus.cc:515] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.463943 11589 leader_election.cc:290] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:46.467878 10013 tablet_bootstrap.cc:492] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:46.469614 9618 leader_election.cc:343] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:46.469466 11564 leader_election.cc:290] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:46.477489 10998 tablet_loader.cc:96] loaded metadata for tablet 644170f17f1b45a19677d958b9745e16 (table default.Table_09fd612a3c4a4303b8f05530e7d49819 [id=eda14d7d5f7d4391a8c470677ffab4ea])
I20260504 13:56:46.449684 11568 consensus_queue.cc:237] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.480232 10998 tablet_loader.cc:96] loaded metadata for tablet 6677a93d01d54f15a506ae1474bcb501 (table default.Table_4842f50414d34bdb8a10802bc059b421 [id=7fc23620ef7342eea52bcce264eb12b3])
I20260504 13:56:46.491925 9678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75a43e11c8b34bd7bcddbf6e9b8f30dc" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:46.492671 9678 raft_consensus.cc:2468] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:46.495659 10998 tablet_loader.cc:96] loaded metadata for tablet 68e92cc3da9b4bcfadb6fa26245f1c99 (table default.Table_18f9a8030cea4cc0abedb81cd5accc59 [id=5c4f226dc33b428e92ca77959e075d7c])
I20260504 13:56:46.497504 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75a43e11c8b34bd7bcddbf6e9b8f30dc" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:46.498016 9828 raft_consensus.cc:2468] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:46.504557 11597 raft_consensus.cc:493] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:46.505126 11597 raft_consensus.cc:515] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.508064 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75a43e11c8b34bd7bcddbf6e9b8f30dc" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:46.508565 9684 raft_consensus.cc:2468] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:46.509536 9768 leader_election.cc:304] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:46.510150 11604 raft_consensus.cc:2804] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:46.510473 11604 raft_consensus.cc:493] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:46.510784 11604 raft_consensus.cc:3060] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:46.517436 11564 raft_consensus.cc:493] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:46.517975 11564 raft_consensus.cc:515] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.519799 11564 leader_election.cc:290] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:46.520751 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c09e9076214405eaa8846fe08158bb3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:46.522030 9618 leader_election.cc:343] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:46.528491 11604 raft_consensus.cc:515] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.530635 11604 leader_election.cc:290] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:46.531612 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75a43e11c8b34bd7bcddbf6e9b8f30dc" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:46.532058 9684 raft_consensus.cc:3060] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:46.539413 9684 raft_consensus.cc:2468] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:46.539970 10013 tablet_bootstrap.cc:654] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:46.540330 9768 leader_election.cc:304] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:46.557670 10998 tablet_loader.cc:96] loaded metadata for tablet 6dd139c347f7406aac340cc6ac994273 (table default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56])
I20260504 13:56:46.564671 9674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df71dc4f2cf84cda8cfc534a0eea86f8" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:46.565249 9674 raft_consensus.cc:2468] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:46.565191 9473 leader_election.cc:304] T 36c8e9ba29a1466082a3e8bedec87b91 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:46.566149 11589 raft_consensus.cc:2749] T 36c8e9ba29a1466082a3e8bedec87b91 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:46.573148 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e29a7365caf04506b15089f10e6c1677" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:46.583271 11597 leader_election.cc:290] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:46.583914 11597 raft_consensus.cc:2804] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:46.584419 11597 raft_consensus.cc:697] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:46.585243 11597 consensus_queue.cc:237] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.474920 9183 catalog_manager.cc:5671] T 99e819494171485e9af8bc8ec33a6ff0 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:46.598130 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df71dc4f2cf84cda8cfc534a0eea86f8" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:46.599018 9931 tablet_bootstrap.cc:492] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:46.600639 9931 ts_tablet_manager.cc:1403] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.164s user 0.017s sys 0.006s
I20260504 13:56:46.602349 9474 leader_election.cc:304] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:46.603660 11432 raft_consensus.cc:2804] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:46.603986 11432 raft_consensus.cc:493] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:46.604305 11432 raft_consensus.cc:3060] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:46.606060 9474 leader_election.cc:343] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:46.612108 11432 raft_consensus.cc:515] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:46.614078 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 128526us
I20260504 13:56:46.618228 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75a43e11c8b34bd7bcddbf6e9b8f30dc" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:46.618901 9533 raft_consensus.cc:2393] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:46.618854 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75a43e11c8b34bd7bcddbf6e9b8f30dc" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:46.619601 9531 raft_consensus.cc:2393] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:46.630178 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddd30e9259a04d59b5c1bb82aabf7952" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:46.639680 11432 leader_election.cc:290] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:46.635783 10998 tablet_loader.cc:96] loaded metadata for tablet 6f4c8ffb2175493a981c18676485ac04 (table default.Table_93e03451ea09440d847c57c026230518 [id=6ad5facbe9ca40eeb7746baef5c566a7])
I20260504 13:56:46.646274 10998 tablet_loader.cc:96] loaded metadata for tablet 75a43e11c8b34bd7bcddbf6e9b8f30dc (table default.Table_09255abca80f41dc895d81ccd89b9c7a [id=24485cf365ad4f9b89fa0f716fdb34cd])
I20260504 13:56:46.650610 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c09e9076214405eaa8846fe08158bb3" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:46.660719 9183 catalog_manager.cc:5671] T 7d27b04d3f4443c5ade9616d8626d50e P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:46.669775 9931 raft_consensus.cc:359] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.670394 9931 raft_consensus.cc:385] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:46.670652 9931 raft_consensus.cc:740] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:46.671941 9931 consensus_queue.cc:260] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.672586 11604 raft_consensus.cc:493] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:46.672987 11604 raft_consensus.cc:515] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.673841 9931 ts_tablet_manager.cc:1434] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.073s user 0.004s sys 0.000s
I20260504 13:56:46.674655 9931 tablet_bootstrap.cc:492] T df71dc4f2cf84cda8cfc534a0eea86f8 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:46.675477 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e1d998c4314db3b3682a0510393749" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:46.675981 9684 raft_consensus.cc:2468] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:46.677558 9768 leader_election.cc:304] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:46.678299 11605 raft_consensus.cc:2804] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:46.678552 11604 leader_election.cc:290] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:46.678620 11605 raft_consensus.cc:493] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:46.678977 11605 raft_consensus.cc:3060] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:46.686491 11605 raft_consensus.cc:515] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.687490 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e1d998c4314db3b3682a0510393749" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:46.717074 11605 leader_election.cc:290] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:46.719499 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75a43e11c8b34bd7bcddbf6e9b8f30dc" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:46.720158 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e1d998c4314db3b3682a0510393749" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:46.720638 9684 raft_consensus.cc:3060] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:46.721876 10013 tablet_bootstrap.cc:492] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:46.722303 10013 ts_tablet_manager.cc:1403] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.255s user 0.011s sys 0.001s
I20260504 13:56:46.725703 10998 tablet_loader.cc:96] loaded metadata for tablet 7989338bc4bf4bad8fb8fe37587d0a16 (table default.Table_10e92469b9cb4e9b965d0a10cfeecca8 [id=99da5017cead4ddeaa3441064b1ed716])
I20260504 13:56:46.726903 10998 tablet_loader.cc:96] loaded metadata for tablet 7a89bac060214d30924398007dfa1fa1 (table default.Table_2f0fc59ee89c402cb8a0fc23341636e6 [id=00b5922717934e15993bafde9f2513e7])
I20260504 13:56:46.732074 10013 raft_consensus.cc:359] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:46.732769 10013 raft_consensus.cc:385] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:46.733120 10013 raft_consensus.cc:740] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:46.734980 10013 consensus_queue.cc:260] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:46.735997 9767 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 54742us
I20260504 13:56:46.736758 10013 ts_tablet_manager.cc:1434] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20260504 13:56:46.737629 10013 tablet_bootstrap.cc:492] T 03e1d998c4314db3b3682a0510393749 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
W20260504 13:56:46.723865 9620 leader_election.cc:343] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:46.740978 9620 leader_election.cc:304] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:46.690233 9931 tablet_bootstrap.cc:654] T df71dc4f2cf84cda8cfc534a0eea86f8 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:46.742913 10013 tablet_bootstrap.cc:654] T 03e1d998c4314db3b3682a0510393749 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:46.706447 11604 raft_consensus.cc:493] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:46.743389 11604 raft_consensus.cc:515] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.746246 9674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99fe335b131b45a587ec1cf526dd7f79" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:46.746879 9674 raft_consensus.cc:2468] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:46.747938 9768 leader_election.cc:304] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:46.748571 11476 raft_consensus.cc:2804] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:46.748942 11476 raft_consensus.cc:493] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:46.749219 11476 raft_consensus.cc:3060] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:46.749341 11604 leader_election.cc:290] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:46.750229 11564 raft_consensus.cc:2749] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:46.727583 9684 raft_consensus.cc:2468] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
W20260504 13:56:46.760394 9620 leader_election.cc:343] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:46.760466 9768 leader_election.cc:304] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:46.760736 9620 leader_election.cc:304] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:46.761281 11564 raft_consensus.cc:2749] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:46.761658 11476 raft_consensus.cc:515] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.763319 11604 raft_consensus.cc:2804] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
W20260504 13:56:46.763453 9620 leader_election.cc:343] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:46.763792 9620 leader_election.cc:304] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:46.764374 11604 raft_consensus.cc:697] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:46.764603 11570 raft_consensus.cc:2749] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:46.765439 11604 consensus_queue.cc:237] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.769733 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99fe335b131b45a587ec1cf526dd7f79" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:46.770227 9684 raft_consensus.cc:3060] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:46.775082 11597 raft_consensus.cc:493] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:46.775246 11476 leader_election.cc:290] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:46.775542 11597 raft_consensus.cc:515] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:46.776701 9767 leader_election.cc:343] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:46.779831 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e1d998c4314db3b3682a0510393749" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:46.780596 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99fe335b131b45a587ec1cf526dd7f79" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:46.662530 9183 catalog_manager.cc:5671] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:46.787829 10998 tablet_loader.cc:96] loaded metadata for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13])
I20260504 13:56:46.788911 10998 tablet_loader.cc:96] loaded metadata for tablet 7d27b04d3f4443c5ade9616d8626d50e (table default.Table_8ccb9fc27b374611b20008514d3ad483 [id=6c295b6b72c94ca0b5f1805a555e7f77])
I20260504 13:56:46.789944 10998 tablet_loader.cc:96] loaded metadata for tablet 7eaa40eb4b1c425a9edfb7eaba953b65 (table default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e])
I20260504 13:56:46.792351 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99fe335b131b45a587ec1cf526dd7f79" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
W20260504 13:56:46.796494 9767 leader_election.cc:343] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:46.797341 9767 leader_election.cc:343] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:46.798029 9767 leader_election.cc:343] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:46.798691 9767 leader_election.cc:343] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Not found: Tablet not found: 41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:46.800843 9183 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
W20260504 13:56:46.804574 9768 leader_election.cc:343] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:46.804929 9768 leader_election.cc:304] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:46.805207 9183 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 3 got 0
I20260504 13:56:46.805550 11597 leader_election.cc:290] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:46.806165 11597 raft_consensus.cc:2749] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:56:46.812492 9620 leader_election.cc:343] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:46.812898 9620 leader_election.cc:304] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:46.813567 11570 raft_consensus.cc:2749] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:46.814567 10998 tablet_loader.cc:96] loaded metadata for tablet 7ed3ee9a13e84594905db47aa33040c2 (table default.Table_6462cb42c34a4eb8976d8d2a4f82d434 [id=6e3151859b6d4da3a477367687862671])
I20260504 13:56:46.816283 10998 tablet_loader.cc:96] loaded metadata for tablet 8702ce3ff1a845f3b70012855cd17efd (table default.Table_24dab2dd1f45402786d80cba80bbad82 [id=80ca58df22214450b894afea377e9c16])
I20260504 13:56:46.816998 9931 tablet_bootstrap.cc:492] T df71dc4f2cf84cda8cfc534a0eea86f8 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:46.817323 10998 tablet_loader.cc:96] loaded metadata for tablet 88619eaa917d4c33a910fc1347ab126e (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c])
I20260504 13:56:46.817503 9931 ts_tablet_manager.cc:1403] T df71dc4f2cf84cda8cfc534a0eea86f8 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.143s user 0.010s sys 0.000s
I20260504 13:56:46.820184 9931 raft_consensus.cc:359] T df71dc4f2cf84cda8cfc534a0eea86f8 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.820562 9473 leader_election.cc:304] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:46.820940 9931 raft_consensus.cc:385] T df71dc4f2cf84cda8cfc534a0eea86f8 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:46.821017 9684 raft_consensus.cc:2468] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:46.821318 9931 raft_consensus.cc:740] T df71dc4f2cf84cda8cfc534a0eea86f8 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:46.822263 11618 raft_consensus.cc:2804] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:46.822634 11618 raft_consensus.cc:493] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:46.822999 11618 raft_consensus.cc:3060] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:46.824455 9931 consensus_queue.cc:260] T df71dc4f2cf84cda8cfc534a0eea86f8 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.837455 9183 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4]). Expected version 1 got 0
I20260504 13:56:46.847886 9768 leader_election.cc:304] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:46.848894 11604 raft_consensus.cc:2804] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:46.848642 11618 raft_consensus.cc:515] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.849325 11604 raft_consensus.cc:697] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:46.850056 11604 consensus_queue.cc:237] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.852548 9931 ts_tablet_manager.cc:1434] T df71dc4f2cf84cda8cfc534a0eea86f8 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.035s user 0.002s sys 0.004s
I20260504 13:56:46.853513 9931 tablet_bootstrap.cc:492] T 6dd139c347f7406aac340cc6ac994273 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:46.854930 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 13:56:46.856881 10690 leader_election.cc:336] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781): Timed out: RequestConsensusVote RPC to 127.8.238.252:44781 timed out after 0.727s (SENT)
I20260504 13:56:46.857273 10690 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:46.868129 11623 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 13:56:46.869400 9736 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:46.883709 9674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75a43e11c8b34bd7bcddbf6e9b8f30dc" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:46.884413 9674 raft_consensus.cc:2393] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 1: Already voted for candidate d4051d2bf0bf410eb8acd107560df888 in this term.
I20260504 13:56:46.881014 10013 tablet_bootstrap.cc:492] T 03e1d998c4314db3b3682a0510393749 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:46.894626 10013 ts_tablet_manager.cc:1403] T 03e1d998c4314db3b3682a0510393749 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.157s user 0.008s sys 0.010s
I20260504 13:56:46.896116 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df71dc4f2cf84cda8cfc534a0eea86f8" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:46.896646 9828 raft_consensus.cc:3060] T df71dc4f2cf84cda8cfc534a0eea86f8 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:46.897392 10013 raft_consensus.cc:359] T 03e1d998c4314db3b3682a0510393749 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.898077 10013 raft_consensus.cc:385] T 03e1d998c4314db3b3682a0510393749 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:46.898355 10013 raft_consensus.cc:740] T 03e1d998c4314db3b3682a0510393749 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:46.899040 10013 consensus_queue.cc:260] T 03e1d998c4314db3b3682a0510393749 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.900761 9931 tablet_bootstrap.cc:654] T 6dd139c347f7406aac340cc6ac994273 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:46.902143 10998 tablet_loader.cc:96] loaded metadata for tablet 89edd71cf5aa4a6abe2084695d3ed807 (table default.Table_69c9551ae2cf4ef399e40017a33bad73 [id=2d9b8dca85e9477eb77877d5618e9b6f])
I20260504 13:56:46.839080 9183 catalog_manager.cc:5671] T cc2f95dadd544f57bd6d1467d30e954d P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:46.905205 9828 raft_consensus.cc:2468] T df71dc4f2cf84cda8cfc534a0eea86f8 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:46.906005 9474 leader_election.cc:304] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:46.906831 11480 raft_consensus.cc:2804] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:46.907557 11480 raft_consensus.cc:697] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:46.908825 11476 raft_consensus.cc:493] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:46.908402 11480 consensus_queue.cc:237] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.909516 11476 raft_consensus.cc:515] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:46.912451 11476 leader_election.cc:290] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:46.915308 10013 ts_tablet_manager.cc:1434] T 03e1d998c4314db3b3682a0510393749 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.020s user 0.003s sys 0.004s
I20260504 13:56:46.916177 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57983fcecdb64a42884f797b4db57df0" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:46.916788 9684 raft_consensus.cc:2468] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:46.917685 9768 leader_election.cc:304] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:46.918763 11476 raft_consensus.cc:2804] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:46.919230 11476 raft_consensus.cc:493] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:46.919940 11476 raft_consensus.cc:3060] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:46.921397 11618 leader_election.cc:290] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:46.923321 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57983fcecdb64a42884f797b4db57df0" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:46.923866 9539 raft_consensus.cc:2468] T 57983fcecdb64a42884f797b4db57df0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:46.924317 9674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df71dc4f2cf84cda8cfc534a0eea86f8" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:46.924854 9674 raft_consensus.cc:3060] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:46.927716 10013 tablet_bootstrap.cc:492] T 99fe335b131b45a587ec1cf526dd7f79 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:46.926887 10998 tablet_loader.cc:96] loaded metadata for tablet 8a985338790e4887af38c49ba013417f (table default.Table_5636294717ff4c63939c2cf9facfc788 [id=6acf451855024161b7826094f8387a9c])
I20260504 13:56:46.930051 11589 raft_consensus.cc:493] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:46.930483 11589 raft_consensus.cc:515] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:46.932618 11589 leader_election.cc:290] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:46.932789 9674 raft_consensus.cc:2468] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:46.935221 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dd139c347f7406aac340cc6ac994273" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:46.936335 9474 leader_election.cc:343] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:46.947500 9183 catalog_manager.cc:5671] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } }
I20260504 13:56:46.948853 9183 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 2ddf890c17964e7f90409ebe3c4034ef (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]). Expected version 2 got 0
I20260504 13:56:46.958660 9881 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:46.971664 9931 tablet_bootstrap.cc:492] T 6dd139c347f7406aac340cc6ac994273 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:46.972066 9931 ts_tablet_manager.cc:1403] T 6dd139c347f7406aac340cc6ac994273 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.119s user 0.002s sys 0.008s
I20260504 13:56:46.979523 9931 raft_consensus.cc:359] T 6dd139c347f7406aac340cc6ac994273 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:46.980115 9931 raft_consensus.cc:385] T 6dd139c347f7406aac340cc6ac994273 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:46.980352 9931 raft_consensus.cc:740] T 6dd139c347f7406aac340cc6ac994273 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:46.980880 9931 consensus_queue.cc:260] T 6dd139c347f7406aac340cc6ac994273 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:46.982772 9931 ts_tablet_manager.cc:1434] T 6dd139c347f7406aac340cc6ac994273 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
W20260504 13:56:46.984297 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 46427.9us
W20260504 13:56:46.984426 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 43994.7us
I20260504 13:56:46.987385 10998 tablet_loader.cc:96] loaded metadata for tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3 (table default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142])
I20260504 13:56:46.989302 10998 tablet_loader.cc:96] loaded metadata for tablet 8d6db45d2e5c41a3b30cede3301694d3 (table default.Table_862ae682dfb44737a72e7c5df628a957 [id=bbef146162cb4c30a3ba42ca2a36fe9f])
I20260504 13:56:46.990425 10998 tablet_loader.cc:96] loaded metadata for tablet 8db7c4732fa048f9950248f45d144c57 (table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62])
I20260504 13:56:46.999058 9473 leader_election.cc:304] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:47.000962 11618 raft_consensus.cc:2749] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260504 13:56:46.995239 10013 tablet_bootstrap.cc:654] T 99fe335b131b45a587ec1cf526dd7f79 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:47.015851 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dd139c347f7406aac340cc6ac994273" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:47.016355 9682 raft_consensus.cc:2468] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:47.021698 9473 leader_election.cc:304] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:47.022504 11617 raft_consensus.cc:2804] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:47.022855 11617 raft_consensus.cc:493] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:47.023161 11617 raft_consensus.cc:3060] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:47.021768 9183 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]). Expected version 1 got 0
I20260504 13:56:47.032491 10998 tablet_loader.cc:96] loaded metadata for tablet 9077fb9bb1544a2cbeb96f0d11c400db (table default.Table_06ecf48a04d94ac8a7e954ff2c260dbf [id=e11be55403d24c8bb35f4708da2e9a78])
I20260504 13:56:47.032457 9183 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]). Expected version 1 got 0
I20260504 13:56:47.034421 10998 tablet_loader.cc:96] loaded metadata for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1])
I20260504 13:56:47.038218 9588 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:47.044142 11617 raft_consensus.cc:515] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:47.054790 9931 tablet_bootstrap.cc:492] T 644170f17f1b45a19677d958b9745e16 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:47.014541 11476 raft_consensus.cc:515] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.057945 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dd139c347f7406aac340cc6ac994273" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:47.058408 9828 raft_consensus.cc:3060] T 6dd139c347f7406aac340cc6ac994273 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:47.059913 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57983fcecdb64a42884f797b4db57df0" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:47.060367 9539 raft_consensus.cc:3060] T 57983fcecdb64a42884f797b4db57df0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:47.066111 9539 raft_consensus.cc:2468] T 57983fcecdb64a42884f797b4db57df0 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:47.067425 10013 tablet_bootstrap.cc:492] T 99fe335b131b45a587ec1cf526dd7f79 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:47.067863 10013 ts_tablet_manager.cc:1403] T 99fe335b131b45a587ec1cf526dd7f79 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.140s user 0.009s sys 0.004s
I20260504 13:56:47.068212 9767 leader_election.cc:304] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:47.070719 11601 raft_consensus.cc:2804] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:47.071131 11601 raft_consensus.cc:697] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:47.070906 10013 raft_consensus.cc:359] T 99fe335b131b45a587ec1cf526dd7f79 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.072000 10013 raft_consensus.cc:385] T 99fe335b131b45a587ec1cf526dd7f79 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:47.071777 11601 consensus_queue.cc:237] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.072263 10013 raft_consensus.cc:740] T 99fe335b131b45a587ec1cf526dd7f79 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:47.072822 10013 consensus_queue.cc:260] T 99fe335b131b45a587ec1cf526dd7f79 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.073755 11617 leader_election.cc:290] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:47.080693 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6dd139c347f7406aac340cc6ac994273" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:47.081209 9682 raft_consensus.cc:3060] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:47.082307 10998 tablet_loader.cc:96] loaded metadata for tablet 975fa1a4975242a388c15fe73abea1c7 (table default.Table_a660204a2583425aa3ecd09c5cc01368 [id=4683a87561f14c049c1ed8eb68dc24d1])
I20260504 13:56:47.089272 9828 raft_consensus.cc:2468] T 6dd139c347f7406aac340cc6ac994273 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:47.091571 10998 tablet_loader.cc:96] loaded metadata for tablet 982f7c53a1f0482e862b53aeeee7cd25 (table default.Table_1e6b4c9e684e45b3b2510d3979c1b5c4 [id=ecfa480ab0ea40008ff4507ed6c5b964])
I20260504 13:56:47.093053 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57983fcecdb64a42884f797b4db57df0" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:47.093636 9684 raft_consensus.cc:3060] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:47.094712 9931 tablet_bootstrap.cc:654] T 644170f17f1b45a19677d958b9745e16 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:47.099634 10998 tablet_loader.cc:96] loaded metadata for tablet 99e819494171485e9af8bc8ec33a6ff0 (table default.Table_89c38f9ba46e48fab397a7f3d5d1b805 [id=e4f6621892874f979d1c579d487cba44])
I20260504 13:56:47.099952 10013 ts_tablet_manager.cc:1434] T 99fe335b131b45a587ec1cf526dd7f79 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.032s user 0.005s sys 0.001s
I20260504 13:56:47.100853 9474 leader_election.cc:304] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:47.101146 10013 tablet_bootstrap.cc:492] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:47.102957 11617 raft_consensus.cc:2804] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:47.103580 11617 raft_consensus.cc:697] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:47.104408 11617 consensus_queue.cc:237] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:47.109452 11476 leader_election.cc:290] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:47.121693 10998 tablet_loader.cc:96] loaded metadata for tablet 99fe335b131b45a587ec1cf526dd7f79 (table default.Table_655e2e8cf3544fbeb0ce33829bb2c8a0 [id=9db1e031520d405ea4989e30d5194911])
I20260504 13:56:47.128093 10998 tablet_loader.cc:96] loaded metadata for tablet 9f3905bfb30e4b03a7fcf0a2f57e2117 (table default.Table_597b4ca3e37849d0aa9c86eed64d0614 [id=5a597e1cae1d4126ae355289aef1ced6])
I20260504 13:56:47.129248 10998 tablet_loader.cc:96] loaded metadata for tablet 9f80ce19e8f9498b98952a371476f185 (table default.Table_2bed5ea8f43149e8bc3b21cd9d1b3846 [id=15994ccc978545058e324152f70c8c3e])
I20260504 13:56:47.135751 10013 tablet_bootstrap.cc:654] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:47.141921 9931 tablet_bootstrap.cc:492] T 644170f17f1b45a19677d958b9745e16 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:47.142313 9931 ts_tablet_manager.cc:1403] T 644170f17f1b45a19677d958b9745e16 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.088s user 0.008s sys 0.000s
I20260504 13:56:47.143501 9684 raft_consensus.cc:2468] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:47.147954 9931 raft_consensus.cc:359] T 644170f17f1b45a19677d958b9745e16 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.148527 9931 raft_consensus.cc:385] T 644170f17f1b45a19677d958b9745e16 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:47.148774 9931 raft_consensus.cc:740] T 644170f17f1b45a19677d958b9745e16 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:47.164460 9931 consensus_queue.cc:260] T 644170f17f1b45a19677d958b9745e16 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.166815 9931 ts_tablet_manager.cc:1434] T 644170f17f1b45a19677d958b9745e16 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260504 13:56:47.131026 10998 tablet_loader.cc:96] loaded metadata for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81])
I20260504 13:56:47.172277 10998 tablet_loader.cc:96] loaded metadata for tablet a26c6a2ffb8746a3941a56c7259412e3 (table default.Table_225ad80b8318432b88a729064c88e8ec [id=a3cca003025745cebf8e7d403d0778cf])
I20260504 13:56:47.149295 9682 raft_consensus.cc:2468] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:47.173404 10998 tablet_loader.cc:96] loaded metadata for tablet a2bfb681e6824e9f9bdcbea23061d57a (table default.Table_e55147137c794be4bff21afce5d36bbc [id=b45733fe3c5547bea00e3967be81ca35])
I20260504 13:56:47.177028 10013 tablet_bootstrap.cc:492] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:47.177417 10013 ts_tablet_manager.cc:1403] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.007s
I20260504 13:56:47.181310 10013 raft_consensus.cc:359] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.181890 10013 raft_consensus.cc:385] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:47.182123 10013 raft_consensus.cc:740] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:47.182686 10013 consensus_queue.cc:260] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.185880 11564 raft_consensus.cc:493] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:47.186308 11564 raft_consensus.cc:515] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.188333 11564 leader_election.cc:290] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:47.189630 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6677a93d01d54f15a506ae1474bcb501" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:47.191013 9618 leader_election.cc:343] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:47.191123 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6677a93d01d54f15a506ae1474bcb501" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:47.175156 10998 tablet_loader.cc:96] loaded metadata for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f])
W20260504 13:56:47.192194 9620 leader_election.cc:343] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:47.192574 9620 leader_election.cc:304] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:47.193507 11564 raft_consensus.cc:2749] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:47.194334 10998 tablet_loader.cc:96] loaded metadata for tablet ab4c185332a24d96bc3d0fddc0d20693 (table default.Table_af058f50341e4f478c970f8cb52bf0d7 [id=57ccd7218d454c8c908fe06b798bd83b])
I20260504 13:56:47.196470 10013 ts_tablet_manager.cc:1434] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260504 13:56:47.205816 9931 tablet_bootstrap.cc:492] T 37b1664312d347288680d97b62c2aecb P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:47.207782 10013 tablet_bootstrap.cc:492] T 9f80ce19e8f9498b98952a371476f185 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:47.211844 9931 tablet_bootstrap.cc:654] T 37b1664312d347288680d97b62c2aecb P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:47.195472 10998 tablet_loader.cc:96] loaded metadata for tablet acd5f7d3f7c545d18fe19b64e4bc1de6 (table default.Table_f7b06aadc0024855b08b56cad8c7b6a3 [id=4efd9dd69847446d9f05700d3bd3f914])
I20260504 13:56:47.218607 10013 tablet_bootstrap.cc:654] T 9f80ce19e8f9498b98952a371476f185 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:47.216243 10998 tablet_loader.cc:96] loaded metadata for tablet af5c5a6cae4e4b6ea3eece2e21a587ec (table default.Table_cdee25533fe54da0a74b421714b0d7c4 [id=81df524d85774544807ba1902be1aae1])
I20260504 13:56:47.231817 11476 raft_consensus.cc:493] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:47.232306 11476 raft_consensus.cc:515] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
W20260504 13:56:47.237358 9767 leader_election.cc:343] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Not found: Tablet not found: 8bb77106b48a45a2aaa938dc474fa8bd
W20260504 13:56:47.238323 9768 leader_election.cc:343] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 8bb77106b48a45a2aaa938dc474fa8bd
I20260504 13:56:47.238658 9768 leader_election.cc:304] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:47.239348 11597 raft_consensus.cc:2749] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:47.236956 11476 leader_election.cc:290] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:47.243471 11480 raft_consensus.cc:493] T 03e1d998c4314db3b3682a0510393749 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:47.243911 11480 raft_consensus.cc:515] T 03e1d998c4314db3b3682a0510393749 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.246033 10998 tablet_loader.cc:96] loaded metadata for tablet b13303d2a14c4187b82da9f25ce9753e (table default.Table_72e3a254561b4cae927e6b6bcacd87b6 [id=f227576f105c4b1fbd459b3638ce684b])
I20260504 13:56:47.246637 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e1d998c4314db3b3682a0510393749" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:47.246811 11480 leader_election.cc:290] T 03e1d998c4314db3b3682a0510393749 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:47.247428 9682 raft_consensus.cc:2393] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 1: Already voted for candidate d4051d2bf0bf410eb8acd107560df888 in this term.
I20260504 13:56:47.247253 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e1d998c4314db3b3682a0510393749" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:47.249542 10998 tablet_loader.cc:96] loaded metadata for tablet b320a37b29844c7593a12000eaa3da43 (table default.Table_2099388b6e2a455295be76a6d072758e [id=63e34a66cd644ab7bede27373e321714])
I20260504 13:56:47.249396 9474 leader_election.cc:304] T 03e1d998c4314db3b3682a0510393749 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:47.251300 10998 tablet_loader.cc:96] loaded metadata for tablet b37a738f2902402ca76e4edebcdba474 (table default.Table_91cc6c38c9ff43dca25814664068b1c6 [id=da5c427e75e74ea1ae3a5961ed27e0de])
I20260504 13:56:47.251328 11480 raft_consensus.cc:3060] T 03e1d998c4314db3b3682a0510393749 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:47.253052 10998 tablet_loader.cc:96] loaded metadata for tablet b6103d953ace4cb8be3c580b3316f87c (table default.Table_a992fb6e038e4408a781d6d8ead37f54 [id=937a02c37dbd45d5bc92979ceaee1c06])
I20260504 13:56:47.266770 10013 tablet_bootstrap.cc:492] T 9f80ce19e8f9498b98952a371476f185 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:47.267241 10013 ts_tablet_manager.cc:1403] T 9f80ce19e8f9498b98952a371476f185 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.000s
I20260504 13:56:47.272194 10013 raft_consensus.cc:359] T 9f80ce19e8f9498b98952a371476f185 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.272869 10013 raft_consensus.cc:385] T 9f80ce19e8f9498b98952a371476f185 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:47.273133 10013 raft_consensus.cc:740] T 9f80ce19e8f9498b98952a371476f185 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:47.273712 10013 consensus_queue.cc:260] T 9f80ce19e8f9498b98952a371476f185 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.277395 9931 tablet_bootstrap.cc:492] T 37b1664312d347288680d97b62c2aecb P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:47.277793 9931 ts_tablet_manager.cc:1403] T 37b1664312d347288680d97b62c2aecb P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.072s user 0.010s sys 0.001s
I20260504 13:56:47.280324 10013 ts_tablet_manager.cc:1434] T 9f80ce19e8f9498b98952a371476f185 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260504 13:56:47.281216 10013 tablet_bootstrap.cc:492] T ddd30e9259a04d59b5c1bb82aabf7952 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:47.298543 9931 raft_consensus.cc:359] T 37b1664312d347288680d97b62c2aecb P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.299162 9931 raft_consensus.cc:385] T 37b1664312d347288680d97b62c2aecb P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:47.299413 9931 raft_consensus.cc:740] T 37b1664312d347288680d97b62c2aecb P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:47.299908 9931 consensus_queue.cc:260] T 37b1664312d347288680d97b62c2aecb P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.303694 11480 raft_consensus.cc:2749] T 03e1d998c4314db3b3682a0510393749 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:47.311270 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:47.324776 9931 ts_tablet_manager.cc:1434] T 37b1664312d347288680d97b62c2aecb P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.047s user 0.005s sys 0.000s
I20260504 13:56:47.325652 9931 tablet_bootstrap.cc:492] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:47.331465 10998 tablet_loader.cc:96] loaded metadata for tablet b7b942037c0e45a6b3cccba288b90b00 (table default.Table_dd11ea89ad7248c39ec114af58cf598f [id=14bd1983ba95468e9e70d8f5724f42e6])
I20260504 13:56:47.335566 11476 consensus_queue.cc:1048] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:47.332609 10998 tablet_loader.cc:96] loaded metadata for tablet b84c3bedc7a84e0d8a873fb15689b314 (table default.Table_7f912e1d4381428e9572cbb4b13f4eec [id=fb18c55d21704b2ab04dbf35d21377bf])
I20260504 13:56:47.341181 10998 tablet_loader.cc:96] loaded metadata for tablet b8e4239942f34fb081a026a35740e465 (table default.Table_11d8ab61a6044154bb599875a0bc9004 [id=73e12524eb9e416782bf9f7fb58c93bd])
I20260504 13:56:47.342227 10998 tablet_loader.cc:96] loaded metadata for tablet b8ffeae4e5af4cc8a3eb3bc0bedf18cc (table default.Table_003cbc6fdf0b4dddaba008af167dc621 [id=4629003a2d8d461c9e309d49ddecea6a])
I20260504 13:56:47.346486 11564 raft_consensus.cc:493] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:47.346910 11564 raft_consensus.cc:515] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.348733 11564 leader_election.cc:290] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:47.349912 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f80ce19e8f9498b98952a371476f185" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:47.350229 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f80ce19e8f9498b98952a371476f185" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:47.350800 9531 raft_consensus.cc:2468] T 9f80ce19e8f9498b98952a371476f185 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
W20260504 13:56:47.351759 9620 leader_election.cc:343] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:47.352818 9618 leader_election.cc:304] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:47.353725 11564 raft_consensus.cc:2804] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:47.354174 11570 raft_consensus.cc:493] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:47.354590 11570 raft_consensus.cc:515] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.356261 11570 leader_election.cc:290] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:47.354061 11564 raft_consensus.cc:493] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:47.357620 11564 raft_consensus.cc:3060] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:47.363866 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd635a863f7a41d087947f619e41f95c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:47.364553 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd635a863f7a41d087947f619e41f95c" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:47.365825 9618 leader_election.cc:343] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:47.366906 9620 leader_election.cc:343] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:47.367307 9620 leader_election.cc:304] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:47.369377 11570 raft_consensus.cc:2749] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:47.384541 11570 raft_consensus.cc:493] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:47.385069 11570 raft_consensus.cc:515] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.388365 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564d0fafb49a4b4bb3bdb240c74c3410" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:47.389704 9618 leader_election.cc:343] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:47.390671 11570 leader_election.cc:290] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:47.395437 10998 tablet_loader.cc:96] loaded metadata for tablet ba4584755aae47e0b0f14c6c55474d18 (table default.Table_2d5a582adb634895a55193dd861f3b60 [id=30f349197d564e8da82855fec17b4cd2])
I20260504 13:56:47.398612 10998 tablet_loader.cc:96] loaded metadata for tablet bae90898739b4303af0608e39e8c0367 (table default.Table_073a6dae9da444bea1f96b2f33fe1a85 [id=39c692c399dd43ca963357b6818871cf])
I20260504 13:56:47.400416 10998 tablet_loader.cc:96] loaded metadata for tablet bd635a863f7a41d087947f619e41f95c (table default.Table_849bbe7902ca44969c5208828774fa06 [id=c146e84e0e844deda9af40562f980047])
I20260504 13:56:47.411556 9931 tablet_bootstrap.cc:654] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:47.420761 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 53878.9us
I20260504 13:56:47.424674 10013 tablet_bootstrap.cc:654] T ddd30e9259a04d59b5c1bb82aabf7952 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:47.424784 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564d0fafb49a4b4bb3bdb240c74c3410" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:47.443682 11617 raft_consensus.cc:493] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:47.444197 11617 raft_consensus.cc:515] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:47.446880 9768 consensus_peers.cc:597] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21d6358b451d4b8d87a22e30e0780f44. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:47.447441 10998 tablet_loader.cc:96] loaded metadata for tablet bdf1c3cef3334a849f7b5e429b7f4990 (table default.Table_42f086bc327c42ccad96925c681a386a [id=2c6aa17153154189a33f5f3aa00b62e9])
I20260504 13:56:47.447520 11617 leader_election.cc:290] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:47.450250 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "644170f17f1b45a19677d958b9745e16" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:47.450775 9829 raft_consensus.cc:2468] T 644170f17f1b45a19677d958b9745e16 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:47.451678 10998 tablet_loader.cc:96] loaded metadata for tablet c1a01231bb044433ae5827d07a6592f8 (table default.Table_3387191fcd6046abbf257447bfbb27f7 [id=d195f5bef6ce49f1b87a4bf5f516eebb])
I20260504 13:56:47.451690 9474 leader_election.cc:304] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:47.452477 11617 raft_consensus.cc:2804] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:47.452804 11617 raft_consensus.cc:493] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:47.453114 10998 tablet_loader.cc:96] loaded metadata for tablet c3112c3c3629412e9b76407c7cba8f92 (table default.Table_b59e5e285d50401e94d51afeb4cbc500 [id=8f16af9aec63405cbae2a91b84fa3009])
I20260504 13:56:47.453121 11617 raft_consensus.cc:3060] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:47.455895 10998 tablet_loader.cc:96] loaded metadata for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a])
I20260504 13:56:47.457751 10998 tablet_loader.cc:96] loaded metadata for tablet c6ebc96408d64c2da7dc6c4fc158995c (table default.Table_5ff6dab0e47d41e681533fca02a36153 [id=230a8885dd2b4995815b650d3d9f9f75])
I20260504 13:56:47.458881 10998 tablet_loader.cc:96] loaded metadata for tablet c8989cdde4ca4e349079596e33723b2c (table default.Table_2d928cb3553f40bc8e825d0741ef3199 [id=51b254be98ee457d892039784e4a5f0b])
I20260504 13:56:47.461390 10998 tablet_loader.cc:96] loaded metadata for tablet cc2f95dadd544f57bd6d1467d30e954d (table default.Table_2585508fc4214c8b82d2957644309ac2 [id=256b605e5a2342ed883ec30e3de0ffda])
I20260504 13:56:47.469193 9931 tablet_bootstrap.cc:492] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:47.469583 9931 ts_tablet_manager.cc:1403] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.144s user 0.005s sys 0.005s
I20260504 13:56:47.469192 11617 raft_consensus.cc:515] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.471951 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "644170f17f1b45a19677d958b9745e16" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:47.472508 9829 raft_consensus.cc:3060] T 644170f17f1b45a19677d958b9745e16 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:47.477437 9931 raft_consensus.cc:359] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.486853 9931 raft_consensus.cc:385] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:47.487514 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "644170f17f1b45a19677d958b9745e16" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:47.487722 9931 raft_consensus.cc:740] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:47.488350 9931 consensus_queue.cc:260] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.420375 11564 raft_consensus.cc:515] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.496673 9683 raft_consensus.cc:2468] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
W20260504 13:56:47.498870 9620 leader_election.cc:343] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:47.499341 9620 leader_election.cc:304] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:47.500243 9931 ts_tablet_manager.cc:1434] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.030s user 0.006s sys 0.001s
I20260504 13:56:47.501073 9931 tablet_bootstrap.cc:492] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:47.493631 11570 raft_consensus.cc:493] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:47.502446 11570 raft_consensus.cc:515] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.503365 11564 leader_election.cc:290] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:47.505472 11570 leader_election.cc:290] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:47.507364 9829 raft_consensus.cc:2468] T 644170f17f1b45a19677d958b9745e16 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:47.507834 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37b1664312d347288680d97b62c2aecb" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:47.508195 9474 leader_election.cc:304] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
W20260504 13:56:47.508970 9618 leader_election.cc:343] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:47.509528 11589 raft_consensus.cc:2804] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:47.509718 10998 tablet_loader.cc:96] loaded metadata for tablet cf11f17e95dc411294c755e17c57811e (table default.Table_2dc0381bd6c84ab59d5bb422de89c0e2 [id=ed756f69850642188b881bd3c680a622])
I20260504 13:56:47.510215 11589 raft_consensus.cc:697] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:47.505765 11564 raft_consensus.cc:2749] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:47.511001 11589 consensus_queue.cc:237] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.513830 10013 tablet_bootstrap.cc:492] T ddd30e9259a04d59b5c1bb82aabf7952 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:47.514223 10013 ts_tablet_manager.cc:1403] T ddd30e9259a04d59b5c1bb82aabf7952 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.233s user 0.010s sys 0.000s
I20260504 13:56:47.515950 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f80ce19e8f9498b98952a371476f185" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:47.516510 9539 raft_consensus.cc:3060] T 9f80ce19e8f9498b98952a371476f185 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:47.517970 9175 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 57983fcecdb64a42884f797b4db57df0 consensus state has no leader
I20260504 13:56:47.520216 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37b1664312d347288680d97b62c2aecb" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:47.520712 9828 raft_consensus.cc:2468] T 37b1664312d347288680d97b62c2aecb P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:47.521906 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f80ce19e8f9498b98952a371476f185" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:47.528549 9931 tablet_bootstrap.cc:654] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:47.529107 10998 tablet_loader.cc:96] loaded metadata for tablet dd3c045379734134b49e94340679c11d (table default.Table_443402c40ae5443daf4ba4258b847303 [id=98126e5c27cd4d3da57fcd88d481de79])
I20260504 13:56:47.541677 10013 raft_consensus.cc:359] T ddd30e9259a04d59b5c1bb82aabf7952 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.542402 10013 raft_consensus.cc:385] T ddd30e9259a04d59b5c1bb82aabf7952 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:47.542685 10013 raft_consensus.cc:740] T ddd30e9259a04d59b5c1bb82aabf7952 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:47.543385 10013 consensus_queue.cc:260] T ddd30e9259a04d59b5c1bb82aabf7952 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.542769 10998 tablet_loader.cc:96] loaded metadata for tablet ddd30e9259a04d59b5c1bb82aabf7952 (table default.Table_512ae07c2df74cb99bb77006d985f2da [id=09959f81c69c49778a6ee33dc1e85aae])
I20260504 13:56:47.544153 9620 leader_election.cc:304] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
W20260504 13:56:47.546027 9620 leader_election.cc:343] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:47.547051 11564 raft_consensus.cc:2804] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:47.547437 11564 raft_consensus.cc:493] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:47.547783 11564 raft_consensus.cc:3060] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:47.550117 10998 tablet_loader.cc:96] loaded metadata for tablet de6714ee86d64112baf13dd1b2b87582 (table default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469])
I20260504 13:56:47.552182 11617 leader_election.cc:290] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:47.557878 11476 consensus_queue.cc:1048] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:47.576339 10998 tablet_loader.cc:96] loaded metadata for tablet df71dc4f2cf84cda8cfc534a0eea86f8 (table default.Table_1611aa9f4dba40e09dae4433e2d20154 [id=2fb2c56e2fb44b9aac476e06e36e5778])
I20260504 13:56:47.579654 9931 tablet_bootstrap.cc:492] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:47.580062 9931 ts_tablet_manager.cc:1403] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.004s
I20260504 13:56:47.582252 9931 raft_consensus.cc:359] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.582830 9931 raft_consensus.cc:385] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:47.593209 10998 tablet_loader.cc:96] loaded metadata for tablet dffddd4b89984c54bc26dcb4a9488525 (table default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568])
I20260504 13:56:47.596344 10998 tablet_loader.cc:96] loaded metadata for tablet e0b9e47b0a4443c5830422b9d0553446 (table default.Table_4207c9e8019c4088b934376c5568d101 [id=3ca89308de104a5fa27cac4c62047304])
W20260504 13:56:47.604030 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 57135us
I20260504 13:56:47.606120 10998 tablet_loader.cc:96] loaded metadata for tablet e0e3b36e1f654a3a8df51480801dabd7 (table default.Table_9e9ef3353dcf4b1fadaa2f51bc8f7041 [id=8dcabd7d67334598926f927a62e630d6])
I20260504 13:56:47.607038 11597 consensus_queue.cc:1048] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:47.617093 9674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "644170f17f1b45a19677d958b9745e16" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:47.617854 9674 raft_consensus.cc:3060] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:47.617841 9539 raft_consensus.cc:2468] T 9f80ce19e8f9498b98952a371476f185 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:47.618019 10013 ts_tablet_manager.cc:1434] T ddd30e9259a04d59b5c1bb82aabf7952 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.104s user 0.006s sys 0.000s
I20260504 13:56:47.618887 10013 tablet_bootstrap.cc:492] T 8d6db45d2e5c41a3b30cede3301694d3 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:47.618896 9618 leader_election.cc:304] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:47.619828 11642 raft_consensus.cc:2804] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:47.620232 11642 raft_consensus.cc:697] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:47.621016 11642 consensus_queue.cc:237] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.628574 11597 consensus_queue.cc:1048] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:47.629078 10998 tablet_loader.cc:96] loaded metadata for tablet e0e70772b7ce4f27a941918aaf66a865 (table default.Table_60fe6ee7a2414e2aa8ea2d57ee4a827d [id=1de44a1bbf06492084cfd70cf540b213])
I20260504 13:56:47.631049 10998 tablet_loader.cc:96] loaded metadata for tablet e29a7365caf04506b15089f10e6c1677 (table default.Table_6b0124eccb02454b9195b8ddaae5bd38 [id=d47a2c6a9cb34ba4ade10fc23913bf7d])
I20260504 13:56:47.645615 10998 tablet_loader.cc:96] loaded metadata for tablet e53381ee0e7f4e9781b833b435bca831 (table default.Table_9b40ba316993413181a9743fb7bab7fb [id=fb4e5ce5526e44de8effdd766f4c257f])
I20260504 13:56:47.646776 10998 tablet_loader.cc:96] loaded metadata for tablet e5d63f665166466dba1e7e36c5e19506 (table default.Table_363d6634e42e4dd6a5686c1f0e401ef2 [id=d0d8b54fb69540589420b87f90b4c823])
I20260504 13:56:47.647315 9674 raft_consensus.cc:2468] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:47.658018 9539 raft_consensus.cc:3060] T 99fe335b131b45a587ec1cf526dd7f79 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:47.658303 10998 tablet_loader.cc:96] loaded metadata for tablet e9e66dacf5794b6195575556be889bce (table default.Table_d6a5a22d78264e799e80d31d5749600a [id=d63acf266ca440729128a717eed32433])
I20260504 13:56:47.672591 10998 tablet_loader.cc:96] loaded metadata for tablet eef975650610423c88ce8e4cfb0b013b (table default.Table_0134ffe7086440ccbcadd076d8fe5d01 [id=693e9b6ed89b4ab7b9d5dc7402e17f20])
I20260504 13:56:47.674104 10998 tablet_loader.cc:96] loaded metadata for tablet f337e9698fe64282a684e90d9c61da5a (table default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12])
I20260504 13:56:47.689512 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
I20260504 13:56:47.675127 10998 tablet_loader.cc:96] loaded metadata for tablet f3e86027e3e74bc196e9a9ffc9fdf8a5 (table default.Table_780e641042db4fd5aeb8af8b1aa08c72 [id=6cb26719d5fe45ccbdef18571c902228])
I20260504 13:56:47.694204 10998 tablet_loader.cc:96] loaded metadata for tablet f71f444ce9214e3f98eff6ac8ca7dde7 (table default.Table_df8874a20e834872a12300a4f5628273 [id=7fc0413d79184a2dbe5c470b8506b4e1])
I20260504 13:56:47.695307 10998 tablet_loader.cc:96] loaded metadata for tablet fac3e36cfa4848a1ab015f89c56c2740 (table default.Table_8cae213b77994b9fba63b8a7c5f27249 [id=df86bccb4ca8414c90f9fd8753724d89])
I20260504 13:56:47.696399 10998 tablet_loader.cc:96] loaded metadata for tablet fe7ab3b577b74a9c9a40cb15c94f8aa1 (table default.Table_dee1335eff89438e9c68835a4c258031 [id=1714df2b31ff4865b6b357500d121a26])
I20260504 13:56:47.699061 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 3 got 0
I20260504 13:56:47.701493 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4]). Expected version 1 got 0
I20260504 13:56:47.583089 9931 raft_consensus.cc:740] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:47.704085 9931 consensus_queue.cc:260] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.706040 9931 ts_tablet_manager.cc:1434] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.126s user 0.004s sys 0.000s
I20260504 13:56:47.706845 9931 tablet_bootstrap.cc:492] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:47.710143 11604 consensus_queue.cc:1048] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:47.664430 11564 raft_consensus.cc:515] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.746512 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37b1664312d347288680d97b62c2aecb" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:47.747434 9828 raft_consensus.cc:3060] T 37b1664312d347288680d97b62c2aecb P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:47.751868 11659 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:47.757459 9828 raft_consensus.cc:2468] T 37b1664312d347288680d97b62c2aecb P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:47.757647 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 2ddf890c17964e7f90409ebe3c4034ef (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]). Expected version 2 got 0
I20260504 13:56:47.759351 11659 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:47.761931 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:47.763494 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]). Expected version 1 got 0
I20260504 13:56:47.764783 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]). Expected version 1 got 0
I20260504 13:56:47.766412 10810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:47.766947 11659 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:56:47.768667 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet e53381ee0e7f4e9781b833b435bca831 (table default.Table_9b40ba316993413181a9743fb7bab7fb [id=fb4e5ce5526e44de8effdd766f4c257f]). Expected version 1 got 0
I20260504 13:56:47.767725 9328 leader_election.cc:304] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec9f4c3a43d84fc9a420dd10723c01a1; no voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda
I20260504 13:56:47.772236 9620 leader_election.cc:304] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:47.766601 11604 raft_consensus.cc:493] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:47.773397 11642 raft_consensus.cc:2804] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:47.773202 11604 raft_consensus.cc:515] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.774372 11642 raft_consensus.cc:697] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:47.775569 11642 consensus_queue.cc:237] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.777196 11604 leader_election.cc:290] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:47.779506 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d6db45d2e5c41a3b30cede3301694d3" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:47.781014 9672 raft_consensus.cc:2468] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:47.782920 9768 leader_election.cc:304] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:47.777817 11659 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 13:56:47.783656 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d6db45d2e5c41a3b30cede3301694d3" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:47.784960 9767 leader_election.cc:343] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:47.786680 11601 raft_consensus.cc:2804] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:47.787045 11601 raft_consensus.cc:493] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:47.787388 11601 raft_consensus.cc:3060] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:47.795367 11601 raft_consensus.cc:515] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.798511 11601 leader_election.cc:290] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:47.798734 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d6db45d2e5c41a3b30cede3301694d3" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:47.799291 9672 raft_consensus.cc:3060] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:47.803766 10998 catalog_manager.cc:1486] Time spent T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Loading table and tablet metadata into memory: real 13.451s user 0.386s sys 0.064s
I20260504 13:56:47.804085 10998 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260504 13:56:47.804733 9345 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 5 candidate_status { last_received { term: 4 index: 367 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:47.812698 11564 leader_election.cc:290] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:47.817174 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d6db45d2e5c41a3b30cede3301694d3" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:47.817302 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37b1664312d347288680d97b62c2aecb" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
W20260504 13:56:47.818444 9618 leader_election.cc:343] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:47.819586 9767 leader_election.cc:343] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:47.820097 11597 consensus_queue.cc:1048] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:47.805466 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 5 candidate_status { last_received { term: 4 index: 355 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
W20260504 13:56:47.826509 9345 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37744 (request call id 8) took 4967 ms (client timeout 616 ms). Trace:
I20260504 13:56:47.806466 9349 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Refusing update from remote peer 806407748d564439807fe5d73db9abda: Log matching property violated. Preceding OpId in replica: term: 3 index: 354. Preceding OpId from leader: term: 4 index: 355. (index mismatch)
I20260504 13:56:47.827447 9349 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43482 (request call id 19) took 11565 ms. Trace:
I20260504 13:56:47.827795 9349 rpcz_store.cc:276] 0504 13:56:36.261986 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:36.262271 (+ 285us) service_pool.cc:224] Handling call
0504 13:56:47.805970 (+11543699us) raft_consensus.cc:1471] Updating replica for 0 ops
0504 13:56:47.827138 (+ 21168us) raft_consensus.cc:1697] Filling consensus response to leader.
0504 13:56:47.827359 (+ 221us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 13:56:47.828879 9349 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37744 (request call id 9) took 4198 ms (client timeout 616 ms). Trace:
I20260504 13:56:47.806913 9352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 4 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1"
I20260504 13:56:47.807602 9351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 4 candidate_status { last_received { term: 3 index: 351 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
W20260504 13:56:47.835729 9351 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37744 (request call id 5) took 11844 ms (client timeout 573 ms). Trace:
W20260504 13:56:47.836045 9351 rpcz_store.cc:269] 0504 13:56:35.991079 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:35.991285 (+ 206us) service_pool.cc:224] Handling call
0504 13:56:47.835644 (+11844359us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:47.808192 9350 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 4 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
W20260504 13:56:47.837414 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 13:56:47.837389 9350 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43482 (request call id 18) took 11760 ms (client timeout 533 ms). Trace:
I20260504 13:56:47.837373 11476 consensus_queue.cc:1048] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:47.837747 9350 rpcz_store.cc:269] 0504 13:56:36.077278 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:36.077481 (+ 203us) service_pool.cc:224] Handling call
0504 13:56:47.837309 (+11759828us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 13:56:47.838557 9350 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37744 (request call id 11) took 1709 ms (client timeout 727 ms). Trace:
I20260504 13:56:47.808768 9348 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 4 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1"
W20260504 13:56:47.839656 9348 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43482 (request call id 20) took 11469 ms (client timeout 678 ms). Trace:
W20260504 13:56:47.839962 9348 rpcz_store.cc:269] 0504 13:56:36.370334 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:36.370508 (+ 174us) service_pool.cc:224] Handling call
0504 13:56:47.839579 (+11469071us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:47.809289 9353 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 4 candidate_status { last_received { term: 3 index: 354 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
W20260504 13:56:47.840973 9353 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43482 (request call id 16) took 12310 ms (client timeout 533 ms). Trace:
W20260504 13:56:47.841317 9353 rpcz_store.cc:269] 0504 13:56:35.530212 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:35.530401 (+ 189us) service_pool.cc:224] Handling call
0504 13:56:47.840893 (+12310492us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:47.809733 9354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 4 candidate_status { last_received { term: 3 index: 351 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
W20260504 13:56:47.842320 9354 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37744 (request call id 4) took 12456 ms (client timeout 573 ms). Trace:
W20260504 13:56:47.842644 9354 rpcz_store.cc:269] 0504 13:56:35.386350 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:35.386494 (+ 144us) service_pool.cc:224] Handling call
0504 13:56:47.842259 (+12455765us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:47.810122 9347 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 5 candidate_status { last_received { term: 4 index: 355 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
W20260504 13:56:47.843916 9347 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37744 (request call id 6) took 9795 ms (client timeout 664 ms). Trace:
W20260504 13:56:47.844245 9347 rpcz_store.cc:269] 0504 13:56:38.048888 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:38.049078 (+ 190us) service_pool.cc:224] Handling call
0504 13:56:47.843849 (+9794771us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 13:56:47.847582 9346 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37744 (request call id 7) took 9101 ms (client timeout 664 ms). Trace:
W20260504 13:56:47.847958 9346 rpcz_store.cc:269] 0504 13:56:38.746167 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:38.746343 (+ 176us) service_pool.cc:224] Handling call
0504 13:56:47.847522 (+9101179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:47.849457 9342 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:35768 (request call id 328) took 12432 ms. Trace:
I20260504 13:56:47.849748 9342 rpcz_store.cc:276] 0504 13:56:35.417152 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:35.421318 (+ 4166us) service_pool.cc:224] Handling call
0504 13:56:47.848991 (+12427673us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b5400030554. stack: 00007fc2747a9c83 00007fc2747a9b80 00007fc27431a574 000000000050e094 000000000050e029 00007fc27b801951 00007fc27b8a13d9 00007fc27d673136 00007fc27d6b27d7 00007fc27b2b8a24 00007fc27b2b89b2 00007fc27b2b88e1 00007fc27b2b885c 00007fc27b2b7b12 00007fc27958bbcc 00007fc27958b006
0504 13:56:47.849289 (+ 298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50618112}
W20260504 13:56:47.826858 9345 rpcz_store.cc:269] 0504 13:56:42.859315 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:42.863062 (+ 3747us) service_pool.cc:224] Handling call
0504 13:56:47.826435 (+4963373us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 13:56:47.851934 9351 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37744 (request call id 10) took 2490 ms (client timeout 727 ms). Trace:
I20260504 13:56:47.854307 11570 raft_consensus.cc:493] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:47.854761 11570 raft_consensus.cc:515] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.856303 9672 raft_consensus.cc:2468] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
W20260504 13:56:47.852267 9351 rpcz_store.cc:269] 0504 13:56:45.361119 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:47.851755 (+2490636us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:47.851861 (+ 106us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 13:56:47.838869 9350 rpcz_store.cc:269] 0504 13:56:46.129438 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:47.838398 (+1708960us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:47.838509 (+ 111us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260504 13:56:47.859696 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f4c8ffb2175493a981c18676485ac04" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:47.829191 9349 rpcz_store.cc:269] 0504 13:56:43.630442 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:47.828686 (+4198244us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:47.828811 (+ 125us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260504 13:56:47.864583 9931 tablet_bootstrap.cc:654] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:47.857689 11570 leader_election.cc:290] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
W20260504 13:56:47.860620 9620 leader_election.cc:343] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:47.875391 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f4c8ffb2175493a981c18676485ac04" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:47.877592 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 272504us
I20260504 13:56:47.878407 9768 leader_election.cc:304] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters: 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:47.879935 11476 raft_consensus.cc:2804] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:47.880402 11476 raft_consensus.cc:697] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:47.881187 11476 consensus_queue.cc:237] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.884351 9352 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 806407748d564439807fe5d73db9abda in term 4.
I20260504 13:56:47.884436 11597 raft_consensus.cc:493] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:47.884832 11597 raft_consensus.cc:515] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.884992 11624 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.056s
I20260504 13:56:47.886677 11597 leader_election.cc:290] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:47.888796 9618 leader_election.cc:343] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:47.889235 9618 leader_election.cc:304] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
W20260504 13:56:47.889405 9352 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43482 (request call id 17) took 12199 ms (client timeout 678 ms). Trace:
I20260504 13:56:47.890275 11642 raft_consensus.cc:2749] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:47.890234 9352 rpcz_store.cc:269] 0504 13:56:35.689838 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:35.690000 (+ 162us) service_pool.cc:224] Handling call
0504 13:56:47.889320 (+12199320us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:47.893762 9674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddd30e9259a04d59b5c1bb82aabf7952" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:47.894394 9674 raft_consensus.cc:2468] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:47.894701 10013 tablet_bootstrap.cc:654] T 8d6db45d2e5c41a3b30cede3301694d3 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:47.897289 9768 leader_election.cc:304] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:47.897495 10998 catalog_manager.cc:1269] Loaded cluster ID: 302d3e241578494980ac3509cf5306fb
I20260504 13:56:47.897744 10998 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260504 13:56:47.898133 11597 raft_consensus.cc:2804] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:47.898516 11597 raft_consensus.cc:493] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:47.898905 11597 raft_consensus.cc:3060] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:47.922058 10013 tablet_bootstrap.cc:492] T 8d6db45d2e5c41a3b30cede3301694d3 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:47.922473 10013 ts_tablet_manager.cc:1403] T 8d6db45d2e5c41a3b30cede3301694d3 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.304s user 0.006s sys 0.006s
I20260504 13:56:47.949458 10013 raft_consensus.cc:359] T 8d6db45d2e5c41a3b30cede3301694d3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.950103 10013 raft_consensus.cc:385] T 8d6db45d2e5c41a3b30cede3301694d3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:47.950350 10013 raft_consensus.cc:740] T 8d6db45d2e5c41a3b30cede3301694d3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:47.950937 10013 consensus_queue.cc:260] T 8d6db45d2e5c41a3b30cede3301694d3 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.952939 10013 ts_tablet_manager.cc:1434] T 8d6db45d2e5c41a3b30cede3301694d3 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260504 13:56:47.953732 10013 tablet_bootstrap.cc:492] T 6323e1c10e7643f8a28f04e285f620e9 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:47.943022 11597 raft_consensus.cc:515] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:47.970331 10013 tablet_bootstrap.cc:654] T 6323e1c10e7643f8a28f04e285f620e9 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:47.981806 11480 consensus_queue.cc:1048] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:47.996815 10013 tablet_bootstrap.cc:492] T 6323e1c10e7643f8a28f04e285f620e9 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:47.997226 10013 ts_tablet_manager.cc:1403] T 6323e1c10e7643f8a28f04e285f620e9 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.003s
I20260504 13:56:48.005215 10013 raft_consensus.cc:359] T 6323e1c10e7643f8a28f04e285f620e9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:48.005820 10013 raft_consensus.cc:385] T 6323e1c10e7643f8a28f04e285f620e9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:48.006074 10013 raft_consensus.cc:740] T 6323e1c10e7643f8a28f04e285f620e9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:48.007043 10998 catalog_manager.cc:1514] Loading token signing keys...
I20260504 13:56:48.008992 10013 consensus_queue.cc:260] T 6323e1c10e7643f8a28f04e285f620e9 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:48.010012 11597 leader_election.cc:290] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:48.010114 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddd30e9259a04d59b5c1bb82aabf7952" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:48.010599 9672 raft_consensus.cc:3060] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:48.017453 9672 raft_consensus.cc:2468] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:48.019506 10013 ts_tablet_manager.cc:1434] T 6323e1c10e7643f8a28f04e285f620e9 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.022s user 0.007s sys 0.001s
I20260504 13:56:48.020311 9768 leader_election.cc:304] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:48.024585 10013 tablet_bootstrap.cc:492] T cf11f17e95dc411294c755e17c57811e P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:48.033838 11476 raft_consensus.cc:2804] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:48.034444 11476 raft_consensus.cc:697] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:48.036067 11476 consensus_queue.cc:237] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.047821 11674 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 806407748d564439807fe5d73db9abda)
I20260504 13:56:48.048257 11674 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:48.050141 11674 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:48.051891 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 5 candidate_status { last_received { term: 4 index: 370 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:48.054589 9349 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 5 candidate_status { last_received { term: 4 index: 370 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:48.056360 10690 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:48.058616 10013 tablet_bootstrap.cc:654] T cf11f17e95dc411294c755e17c57811e P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:48.061997 11670 consensus_queue.cc:1048] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:48.065693 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddd30e9259a04d59b5c1bb82aabf7952" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:48.066201 9931 tablet_bootstrap.cc:492] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:48.066615 9931 ts_tablet_manager.cc:1403] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.360s user 0.010s sys 0.000s
I20260504 13:56:48.068786 9931 raft_consensus.cc:359] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.085467 9931 raft_consensus.cc:385] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:48.085803 9931 raft_consensus.cc:740] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:48.087400 11674 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 13:56:48.066226 9539 raft_consensus.cc:2468] T ddd30e9259a04d59b5c1bb82aabf7952 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:48.095777 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddd30e9259a04d59b5c1bb82aabf7952" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:48.096474 9539 raft_consensus.cc:3060] T ddd30e9259a04d59b5c1bb82aabf7952 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:48.102876 10998 catalog_manager.cc:6055] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1: Loaded TSK: 0
I20260504 13:56:48.114688 9931 consensus_queue.cc:260] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.123669 11664 consensus_queue.cc:1048] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:48.125996 9931 ts_tablet_manager.cc:1434] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.059s user 0.004s sys 0.005s
I20260504 13:56:48.127007 9931 tablet_bootstrap.cc:492] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:48.135334 11604 consensus_queue.cc:1048] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:48.139853 10998 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260504 13:56:48.157807 9931 tablet_bootstrap.cc:654] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:48.168962 9539 raft_consensus.cc:2468] T ddd30e9259a04d59b5c1bb82aabf7952 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:48.171448 11660 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:48.172210 11660 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:56:48.173660 11659 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 806407748d564439807fe5d73db9abda. Latest consensus state: current_term: 4 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:48.174296 11659 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:56:48.178956 11433 consensus_queue.cc:1048] T df71dc4f2cf84cda8cfc534a0eea86f8 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:48.197791 9931 tablet_bootstrap.cc:492] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:48.199544 9931 ts_tablet_manager.cc:1403] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.005s
I20260504 13:56:48.202316 9931 raft_consensus.cc:359] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:48.205847 9931 raft_consensus.cc:385] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:48.206526 9931 raft_consensus.cc:740] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:48.207732 9188 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4]). Expected version 1 got 0
I20260504 13:56:48.209116 9188 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
I20260504 13:56:48.220914 11570 raft_consensus.cc:493] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:48.221339 11570 raft_consensus.cc:515] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.223060 11570 leader_election.cc:290] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:48.224901 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f337e9698fe64282a684e90d9c61da5a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:48.225956 9618 leader_election.cc:343] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:48.239658 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f337e9698fe64282a684e90d9c61da5a" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:48.240751 9620 leader_election.cc:343] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:48.241133 9620 leader_election.cc:304] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:48.242053 11570 raft_consensus.cc:2749] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:48.267709 9188 catalog_manager.cc:5671] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:48.271787 11570 raft_consensus.cc:493] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:48.272081 9188 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 2ddf890c17964e7f90409ebe3c4034ef (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]). Expected version 2 got 0
I20260504 13:56:48.272174 11570 raft_consensus.cc:515] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.272833 9188 catalog_manager.cc:5671] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:48.273828 11570 leader_election.cc:290] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:48.273949 9736 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:48.274199 9188 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet e53381ee0e7f4e9781b833b435bca831 (table default.Table_9b40ba316993413181a9743fb7bab7fb [id=fb4e5ce5526e44de8effdd766f4c257f]). Expected version 1 got 0
I20260504 13:56:48.275652 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e70772b7ce4f27a941918aaf66a865" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:48.276850 9618 leader_election.cc:343] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:48.278380 9829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e70772b7ce4f27a941918aaf66a865" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:48.279611 9620 leader_election.cc:343] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:48.280049 9620 leader_election.cc:304] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:48.282362 11570 raft_consensus.cc:2749] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:48.284339 9881 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:48.261931 10013 tablet_bootstrap.cc:492] T cf11f17e95dc411294c755e17c57811e P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:48.284752 11480 consensus_queue.cc:1048] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:48.285095 10013 ts_tablet_manager.cc:1403] T cf11f17e95dc411294c755e17c57811e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.261s user 0.012s sys 0.011s
W20260504 13:56:48.286610 10998 catalog_manager.cc:1419] Failed waiting for node to catch up after master election: Illegal state: Node ec9f4c3a43d84fc9a420dd10723c01a1 not leader. Raft Consensus state: current_term: 4 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
W20260504 13:56:48.292037 9329 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:48.224001 9931 consensus_queue.cc:260] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:48.303432 11476 raft_consensus.cc:493] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:48.308050 10013 raft_consensus.cc:359] T cf11f17e95dc411294c755e17c57811e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.308668 10013 raft_consensus.cc:385] T cf11f17e95dc411294c755e17c57811e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:48.308912 10013 raft_consensus.cc:740] T cf11f17e95dc411294c755e17c57811e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:48.303985 11476 raft_consensus.cc:515] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:48.324324 11476 leader_election.cc:290] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:35.468991 10814 rpcz_store.cc:276] 0504 13:56:33.754323 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:33.754439 (+ 116us) service_pool.cc:224] Handling call
0504 13:56:33.922313 (+167874us) raft_consensus.cc:1471] Updating replica for 347 ops
0504 13:56:34.281612 (+359299us) raft_consensus.cc:1534] Early marking committed up to index 0
0504 13:56:34.281666 (+ 54us) raft_consensus.cc:1539] Triggering prepare for 347 ops
0504 13:56:34.282334 (+ 668us) write_op.cc:270] Start()
0504 13:56:34.282447 (+ 113us) write_op.cc:276] Timestamp: P: 1777902968687974 usec, L: 0
0504 13:56:34.331558 (+ 49111us) write_op.cc:270] Start()
0504 13:56:34.331666 (+ 108us) write_op.cc:276] Timestamp: P: 1777902968705321 usec, L: 0
0504 13:56:34.332257 (+ 591us) write_op.cc:270] Start()
0504 13:56:34.334793 (+ 2536us) write_op.cc:276] Timestamp: P: 1777902968707510 usec, L: 0
0504 13:56:34.335369 (+ 576us) write_op.cc:270] Start()
0504 13:56:34.335448 (+ 79us) write_op.cc:276] Timestamp: P: 1777902968712206 usec, L: 0
0504 13:56:34.342702 (+ 7254us) write_op.cc:270] Start()
0504 13:56:34.342789 (+ 87us) write_op.cc:276] Timestamp: P: 1777902968735273 usec, L: 0
0504 13:56:34.343296 (+ 507us) write_op.cc:270] Start()
0504 13:56:34.343392 (+ 96us) write_op.cc:276] Timestamp: P: 1777902968749314 usec, L: 0
0504 13:56:34.343866 (+ 474us) write_op.cc:270] Start()
0504 13:56:34.344005 (+ 139us) write_op.cc:276] Timestamp: P: 1777902968805954 usec, L: 0
0504 13:56:34.357035 (+ 13030us) write_op.cc:270] Start()
0504 13:56:34.357118 (+ 83us) write_op.cc:276] Timestamp: P: 1777902968809473 usec, L: 0
0504 13:56:34.357613 (+ 495us) write_op.cc:270] Start()
0504 13:56:34.357692 (+ 79us) write_op.cc:276] Timestamp: P: 1777902968812665 usec, L: 0
0504 13:56:34.358224 (+ 532us) write_op.cc:270] Start()
0504 13:56:34.358309 (+ 85us) write_op.cc:276] Timestamp: P: 1777902968854805 usec, L: 0
0504 13:56:34.358778 (+ 469us) write_op.cc:270] Start()
0504 13:56:34.358875 (+ 97us) write_op.cc:276] Timestamp: P: 1777902968933040 usec, L: 0
0504 13:56:34.359347 (+ 472us) write_op.cc:270] Start()
0504 13:56:34.359449 (+ 102us) write_op.cc:276] Timestamp: P: 1777902968971329 usec, L: 0
0504 13:56:34.359908 (+ 459us) write_op.cc:270] Start()
0504 13:56:34.359984 (+ 76us) write_op.cc:276] Timestamp: P: 1777902968977367 usec, L: 0
0504 13:56:34.360549 (+ 565us) write_op.cc:270] Start()
0504 13:56:34.360623 (+ 74us) write_op.cc:276] Timestamp: P: 1777902968997191 usec, L: 0
0504 13:56:34.361081 (+ 458us) write_op.cc:270] Start()
0504 13:56:34.361159 (+ 78us) write_op.cc:276] Timestamp: P: 1777902969031174 usec, L: 0
0504 13:56:34.361575 (+ 416us) write_op.cc:270] Start()
0504 13:56:34.361649 (+ 74us) write_op.cc:276] Timestamp: P: 1777902969044515 usec, L: 0
0504 13:56:34.362126 (+ 477us) write_op.cc:270] Start()
0504 13:56:34.362199 (+ 73us) write_op.cc:276] Timestamp: P: 1777902969064406 usec, L: 0
0504 13:56:34.362682 (+ 483us) write_op.cc:270] Start()
0504 13:56:34.362905 (+ 223us) write_op.cc:276] Timestamp: P: 1777902969083296 usec, L: 0
0504 13:56:34.363343 (+ 438us) write_op.cc:270] Start()
0504 13:56:34.363426 (+ 83us) write_op.cc:276] Timestamp: P: 1777902969086644 usec, L: 0
0504 13:56:34.363894 (+ 468us) write_op.cc:270] Start()
0504 13:56:34.363978 (+ 84us) write_op.cc:276] Timestamp: P: 1777902969112913 usec, L: 0
0504 13:56:34.364532 (+ 554us) write_op.cc:270] Start()
0504 13:56:34.364617 (+ 85us) write_op.cc:276] Timestamp: P: 1777902969169522 usec, L: 0
0504 13:56:34.365096 (+ 479us) write_op.cc:270] Start()
0504 13:56:34.365174 (+ 78us) write_op.cc:276] Timestamp: P: 1777902969189217 usec, L: 0
0504 13:56:34.365617 (+ 443us) write_op.cc:270] Start()
0504 13:56:34.365698 (+ 81us) write_op.cc:276] Timestamp: P: 1777902969202421 usec, L: 0
0504 13:56:34.366142 (+ 444us) write_op.cc:270] Start()
0504 13:56:34.366261 (+ 119us) write_op.cc:276] Timestamp: P: 1777902969259837 usec, L: 0
0504 13:56:34.366858 (+ 597us) write_op.cc:270] Start()
0504 13:56:34.366950 (+ 92us) write_op.cc:276] Timestamp: P: 1777902969284712 usec, L: 0
0504 13:56:34.367424 (+ 474us) write_op.cc:270] Start()
0504 13:56:34.367502 (+ 78us) write_op.cc:276] Timestamp: P: 1777902969336922 usec, L: 0
0504 13:56:34.367971 (+ 469us) write_op.cc:270] Start()
0504 13:56:34.368048 (+ 77us) write_op.cc:276] Timestamp: P: 1777902969343994 usec, L: 0
0504 13:56:34.368486 (+ 438us) write_op.cc:270] Start()
0504 13:56:34.368565 (+ 79us) write_op.cc:276] Timestamp: P: 1777902969426557 usec, L: 0
0504 13:56:34.369027 (+ 462us) write_op.cc:270] Start()
0504 13:56:34.369103 (+ 76us) write_op.cc:276] Timestamp: P: 1777902969457910 usec, L: 0
0504 13:56:34.369557 (+ 454us) write_op.cc:270] Start()
0504 13:56:34.369631 (+ 74us) write_op.cc:276] Timestamp: P: 1777902969540767 usec, L: 0
0504 13:56:34.370116 (+ 485us) write_op.cc:270] Start()
0504 13:56:34.370198 (+ 82us) write_op.cc:276] Timestamp: P: 1777902969628768 usec, L: 0
0504 13:56:34.370641 (+ 443us) write_op.cc:270] Start()
0504 13:56:34.370725 (+ 84us) write_op.cc:276] Timestamp: P: 1777902969640960 usec, L: 0
0504 13:56:34.371275 (+ 550us) write_op.cc:270] Start()
0504 13:56:34.371352 (+ 77us) write_op.cc:276] Timestamp: P: 1777902969666671 usec, L: 0
0504 13:56:34.371774 (+ 422us) write_op.cc:270] Start()
0504 13:56:34.371871 (+ 97us) write_op.cc:276] Timestamp: P: 1777902969704443 usec, L: 0
0504 13:56:34.372308 (+ 437us) write_op.cc:270] Start()
0504 13:56:34.372375 (+ 67us) write_op.cc:276] Timestamp: P: 1777902969765387 usec, L: 0
0504 13:56:34.372778 (+ 403us) write_op.cc:270] Start()
0504 13:56:34.372857 (+ 79us) write_op.cc:276] Timestamp: P: 1777902969785493 usec, L: 0
0504 13:56:34.373284 (+ 427us) write_op.cc:270] Start()
0504 13:56:34.373357 (+ 73us) write_op.cc:276] Timestamp: P: 1777902969809029 usec, L: 0
0504 13:56:34.373796 (+ 439us) write_op.cc:270] Start()
0504 13:56:34.373878 (+ 82us) write_op.cc:276] Timestamp: P: 1777902969851168 usec, L: 0
0504 13:56:34.374306 (+ 428us) write_op.cc:270] Start()
0504 13:56:34.374385 (+ 79us) write_op.cc:276] Timestamp: P: 1777902969886371 usec, L: 0
0504 13:56:34.374791 (+ 406us) write_op.cc:270] Start()
0504 13:56:34.374875 (+ 84us) write_op.cc:276] Timestamp: P: 1777902969916777 usec, L: 0
0504 13:56:34.375325 (+ 450us) write_op.cc:270] Start()
0504 13:56:34.375397 (+ 72us) write_op.cc:276] Timestamp: P: 1777902969932369 usec, L: 0
0504 13:56:34.375831 (+ 434us) write_op.cc:270] Start()
0504 13:56:34.375905 (+ 74us) write_op.cc:276] Timestamp: P: 1777902969950658 usec, L: 0
0504 13:56:34.376349 (+ 444us) write_op.cc:270] Start()
0504 13:56:34.376430 (+ 81us) write_op.cc:276] Timestamp: P: 1777902969994289 usec, L: 0
0504 13:56:34.376856 (+ 426us) write_op.cc:270] Start()
0504 13:56:34.376929 (+ 73us) write_op.cc:276] Timestamp: P: 1777902970085511 usec, L: 0
0504 13:56:34.377355 (+ 426us) write_op.cc:270] Start()
0504 13:56:34.377438 (+ 83us) write_op.cc:276] Timestamp: P: 1777902970168235 usec, L: 0
0504 13:56:34.378020 (+ 582us) write_op.cc:270] Start()
0504 13:56:34.378096 (+ 76us) write_op.cc:276] Timestamp: P: 1777902970240434 usec, L: 0
0504 13:56:34.378533 (+ 437us) write_op.cc:270] Start()
0504 13:56:34.378608 (+ 75us) write_op.cc:276] Timestamp: P: 1777902970269407 usec, L: 0
0504 13:56:34.379012 (+ 404us) write_op.cc:270] Start()
0504 13:56:34.379145 (+ 133us) write_op.cc:276] Timestamp: P: 1777902970304645 usec, L: 0
0504 13:56:34.379679 (+ 534us) write_op.cc:270] Start()
0504 13:56:34.379750 (+ 71us) write_op.cc:276] Timestamp: P: 1777902970306423 usec, L: 0
0504 13:56:34.380141 (+ 391us) write_op.cc:270] Start()
0504 13:56:34.380210 (+ 69us) write_op.cc:276] Timestamp: P: 1777902970308491 usec, L: 0
0504 13:56:34.380621 (+ 411us) write_op.cc:270] Start()
0504 13:56:34.380688 (+ 67us) write_op.cc:276] Timestamp: P: 1777902970336253 usec, L: 0
0504 13:56:34.381107 (+ 419us) write_op.cc:270] Start()
0504 13:56:34.381192 (+ 85us) write_op.cc:276] Timestamp: P: 1777902970508788 usec, L: 0
0504 13:56:34.381601 (+ 409us) write_op.cc:270] Start()
0504 13:56:34.381669 (+ 68us) write_op.cc:276] Timestamp: P: 1777902970516612 usec, L: 0
0504 13:56:34.382070 (+ 401us) write_op.cc:270] Start()
0504 13:56:34.382138 (+ 68us) write_op.cc:276] Timestamp: P: 1777902970524671 usec, L: 0
0504 13:56:34.382528 (+ 390us) write_op.cc:270] Start()
0504 13:56:34.382601 (+ 73us) write_op.cc:276] Timestamp: P: 1777902970530638 usec, L: 0
0504 13:56:34.382995 (+ 394us) write_op.cc:270] Start()
0504 13:56:34.383066 (+ 71us) write_op.cc:276] Timestamp: P: 1777902970661608 usec, L: 0
0504 13:56:34.383498 (+ 432us) write_op.cc:270] Start()
0504 13:56:34.383566 (+ 68us) write_op.cc:276] Timestamp: P: 1777902970777084 usec, L: 0
0504 13:56:34.383970 (+ 404us) write_op.cc:270] Start()
0504 13:56:34.384049 (+ 79us) write_op.cc:276] Timestamp: P: 1777902970811396 usec, L: 0
0504 13:56:34.384457 (+ 408us) write_op.cc:270] Start()
0504 13:56:34.384525 (+ 68us) write_op.cc:276] Timestamp: P: 1777902970826567 usec, L: 0
0504 13:56:34.384914 (+ 389us) write_op.cc:270] Start()
0504 13:56:34.385055 (+ 141us) write_op.cc:276] Timestamp: P: 1777902970841491 usec, L: 0
0504 13:56:34.385463 (+ 408us) write_op.cc:270] Start()
0504 13:56:34.385536 (+ 73us) write_op.cc:276] Timestamp: P: 1777902970876506 usec, L: 0
0504 13:56:34.385955 (+ 419us) write_op.cc:270] Start()
0504 13:56:34.386026 (+ 71us) write_op.cc:276] Timestamp: P: 1777902970917117 usec, L: 0
0504 13:56:34.386482 (+ 456us) write_op.cc:270] Start()
0504 13:56:34.386582 (+ 100us) write_op.cc:276] Timestamp: P: 1777902970946663 usec, L: 0
0504 13:56:34.387010 (+ 428us) write_op.cc:270] Start()
0504 13:56:34.387081 (+ 71us) write_op.cc:276] Timestamp: P: 1777902971025899 usec, L: 0
0504 13:56:34.387698 (+ 617us) write_op.cc:270] Start()
0504 13:56:34.387766 (+ 68us) write_op.cc:276] Timestamp: P: 1777902971108690 usec, L: 0
0504 13:56:34.388332 (+ 566us) write_op.cc:270] Start()
0504 13:56:34.388402 (+ 70us) write_op.cc:276] Timestamp: P: 1777902971111683 usec, L: 0
0504 13:56:34.388827 (+ 425us) write_op.cc:270] Start()
0504 13:56:34.388898 (+ 71us) write_op.cc:276] Timestamp: P: 1777902971187359 usec, L: 0
0504 13:56:34.389316 (+ 418us) write_op.cc:270] Start()
0504 13:56:34.389386 (+ 70us) write_op.cc:276] Timestamp: P: 1777902971214451 usec, L: 0
0504 13:56:34.389808 (+ 422us) write_op.cc:270] Start()
0504 13:56:34.389893 (+ 85us) write_op.cc:276] Timestamp: P: 1777902971216988 usec, L: 0
0504 13:56:34.390314 (+ 421us) write_op.cc:270] Start()
0504 13:56:34.390389 (+ 75us) write_op.cc:276] Timestamp: P: 1777902971218648 usec, L: 0
0504 13:56:34.390845 (+ 456us) write_op.cc:270] Start()
0504 13:56:34.390930 (+ 85us) write_op.cc:276] Timestamp: P: 1777902971264681 usec, L: 0
0504 13:56:34.391371 (+ 441us) write_op.cc:270] Start()
0504 13:56:34.391444 (+ 73us) write_op.cc:276] Timestamp: P: 1777902971274978 usec, L: 0
0504 13:56:34.391898 (+ 454us) write_op.cc:270] Start()
0504 13:56:34.391970 (+ 72us) write_op.cc:276] Timestamp: P: 1777902971365472 usec, L: 0
0504 13:56:34.392397 (+ 427us) write_op.cc:270] Start()
0504 13:56:34.392472 (+ 75us) write_op.cc:276] Timestamp: P: 1777902971393292 usec, L: 0
0504 13:56:34.392895 (+ 423us) write_op.cc:270] Start()
0504 13:56:34.392965 (+ 70us) write_op.cc:276] Timestamp: P: 1777902971397827 usec, L: 0
0504 13:56:34.393371 (+ 406us) write_op.cc:270] Start()
0504 13:56:34.393445 (+ 74us) write_op.cc:276] Timestamp: P: 1777902971432848 usec, L: 0
0504 13:56:34.393873 (+ 428us) write_op.cc:270] Start()
0504 13:56:34.393943 (+ 70us) write_op.cc:276] Timestamp: P: 1777902971463430 usec, L: 0
0504 13:56:34.394340 (+ 397us) write_op.cc:270] Start()
0504 13:56:34.394409 (+ 69us) write_op.cc:276] Timestamp: P: 1777902971511872 usec, L: 0
0504 13:56:34.394826 (+ 417us) write_op.cc:270] Start()
0504 13:56:34.394895 (+ 69us) write_op.cc:276] Timestamp: P: 1777902971513860 usec, L: 0
0504 13:56:34.395312 (+ 417us) write_op.cc:270] Start()
0504 13:56:34.395382 (+ 70us) write_op.cc:276] Timestamp: P: 1777902971562281 usec, L: 0
0504 13:56:34.395783 (+ 401us) write_op.cc:270] Start()
0504 13:56:34.395863 (+ 80us) write_op.cc:276] Timestamp: P: 1777902971605703 usec, L: 0
0504 13:56:34.406147 (+ 10284us) write_op.cc:270] Start()
0504 13:56:34.406247 (+ 100us) write_op.cc:276] Timestamp: P: 1777902971659718 usec, L: 0
0504 13:56:34.406790 (+ 543us) write_op.cc:270] Start()
0504 13:56:34.406898 (+ 108us) write_op.cc:276] Timestamp: P: 1777902971715974 usec, L: 0
0504 13:56:34.407418 (+ 520us) write_op.cc:270] Start()
0504 13:56:34.407495 (+ 77us) write_op.cc:276] Timestamp: P: 1777902971788698 usec, L: 0
0504 13:56:34.407917 (+ 422us) write_op.cc:270] Start()
0504 13:56:34.407987 (+ 70us) write_op.cc:276] Timestamp: P: 1777902971843032 usec, L: 0
0504 13:56:34.408381 (+ 394us) write_op.cc:270] Start()
0504 13:56:34.408453 (+ 72us) write_op.cc:276] Timestamp: P: 1777902971944325 usec, L: 0
0504 13:56:34.426987 (+ 18534us) write_op.cc:270] Start()
0504 13:56:34.427083 (+ 96us) write_op.cc:276] Timestamp: P: 1777902971972734 usec, L: 0
0504 13:56:34.427617 (+ 534us) write_op.cc:270] Start()
0504 13:56:34.427691 (+ 74us) write_op.cc:276] Timestamp: P: 1777902971995706 usec, L: 0
0504 13:56:34.428145 (+ 454us) write_op.cc:270] Start()
0504 13:56:34.428231 (+ 86us) write_op.cc:276] Timestamp: P: 1777902972029496 usec, L: 0
0504 13:56:34.428652 (+ 421us) write_op.cc:270] Start()
0504 13:56:34.428725 (+ 73us) write_op.cc:276] Timestamp: P: 1777902972078919 usec, L: 0
0504 13:56:34.429149 (+ 424us) write_op.cc:270] Start()
0504 13:56:34.429223 (+ 74us) write_op.cc:276] Timestamp: P: 1777902972087537 usec, L: 0
0504 13:56:34.429628 (+ 405us) write_op.cc:270] Start()
0504 13:56:34.429698 (+ 70us) write_op.cc:276] Timestamp: P: 1777902972111574 usec, L: 0
0504 13:56:34.430127 (+ 429us) write_op.cc:270] Start()
0504 13:56:34.430206 (+ 79us) write_op.cc:276] Timestamp: P: 1777902972125888 usec, L: 0
0504 13:56:34.430605 (+ 399us) write_op.cc:270] Start()
0504 13:56:34.430677 (+ 72us) write_op.cc:276] Timestamp: P: 1777902972180016 usec, L: 0
0504 13:56:34.431644 (+ 967us) write_op.cc:270] Start()
0504 13:56:34.431723 (+ 79us) write_op.cc:276] Timestamp: P: 1777902972213308 usec, L: 0
0504 13:56:34.432265 (+ 542us) write_op.cc:270] Start()
0504 13:56:34.432344 (+ 79us) write_op.cc:276] Timestamp: P: 1777902972242058 usec, L: 0
0504 13:56:34.432776 (+ 432us) write_op.cc:270] Start()
0504 13:56:34.432866 (+ 90us) write_op.cc:276] Timestamp: P: 1777902972259559 usec, L: 0
0504 13:56:34.433310 (+ 444us) write_op.cc:270] Start()
0504 13:56:34.433501 (+ 191us) write_op.cc:276] Timestamp: P: 1777902972391727 usec, L: 0
0504 13:56:34.434076 (+ 575us) write_op.cc:270] Start()
0504 13:56:34.434150 (+ 74us) write_op.cc:276] Timestamp: P: 1777902972394509 usec, L: 0
0504 13:56:34.434519 (+ 369us) write_op.cc:270] Start()
0504 13:56:34.434580 (+ 61us) write_op.cc:276] Timestamp: P: 1777902972458998 usec, L: 0
0504 13:56:34.434939 (+ 359us) write_op.cc:270] Start()
0504 13:56:34.435003 (+ 64us) write_op.cc:276] Timestamp: P: 1777902972488336 usec, L: 0
0504 13:56:34.435385 (+ 382us) write_op.cc:270] Start()
0504 13:56:34.435444 (+ 59us) write_op.cc:276] Timestamp: P: 1777902972498321 usec, L: 0
0504 13:56:34.435824 (+ 380us) write_op.cc:270] Start()
0504 13:56:34.435885 (+ 61us) write_op.cc:276] Timestamp: P: 1777902972520071 usec, L: 0
0504 13:56:34.436227 (+ 342us) write_op.cc:270] Start()
0504 13:56:34.436286 (+ 59us) write_op.cc:276] Timestamp: P: 1777902972524220 usec, L: 0
0504 13:56:34.436657 (+ 371us) write_op.cc:270] Start()
0504 13:56:34.436720 (+ 63us) write_op.cc:276] Timestamp: P: 1777902972526842 usec, L: 0
0504 13:56:34.437118 (+ 398us) write_op.cc:270] Start()
0504 13:56:34.437192 (+ 74us) write_op.cc:276] Timestamp: P: 1777902972625036 usec, L: 0
0504 13:56:34.443128 (+ 5936us) spinlock_profiling.cc:243] Waited 4.11 ms on lock 0x7b5400010280. stack: 00007f4beb944c83 00007f4beb944b80 00007f4beb4b5574 000000000050e094 000000000050e029 00007f4bf2dd5671 00007f4bf2ea9ff9 00007f4bf2e9fc1d 00007f4bf2e63657 00007f4bf2e6312f 00007f4bf2a2a060 00007f4bf2a337c6 00007f4bf2a32c1e 00007f4bf3a64ad9 00007f4bf2115d34 00007f4bf2115cc2
0504 13:56:34.443521 (+ 393us) write_op.cc:270] Start()
0504 13:56:34.443586 (+ 65us) write_op.cc:276] Timestamp: P: 1777902972648352 usec, L: 0
0504 13:56:34.445055 (+ 1469us) write_op.cc:270] Start()
0504 13:56:34.445117 (+ 62us) write_op.cc:276] Timestamp: P: 1777902972676007 usec, L: 0
0504 13:56:34.445633 (+ 516us) write_op.cc:270] Start()
0504 13:56:34.445701 (+ 68us) write_op.cc:276] Timestamp: P: 1777902972725587 usec, L: 0
0504 13:56:34.446225 (+ 524us) write_op.cc:270] Start()
0504 13:56:34.446287 (+ 62us) write_op.cc:276] Timestamp: P: 1777902972735902 usec, L: 0
0504 13:56:34.446667 (+ 380us) write_op.cc:270] Start()
0504 13:56:34.446725 (+ 58us) write_op.cc:276] Timestamp: P: 1777902972743314 usec, L: 0
0504 13:56:34.447069 (+ 344us) write_op.cc:270] Start()
0504 13:56:34.447127 (+ 58us) write_op.cc:276] Timestamp: P: 1777902972768275 usec, L: 0
0504 13:56:34.447506 (+ 379us) write_op.cc:270] Start()
0504 13:56:34.447643 (+ 137us) write_op.cc:276] Timestamp: P: 1777902972814780 usec, L: 0
0504 13:56:34.448017 (+ 374us) write_op.cc:270] Start()
0504 13:56:34.448079 (+ 62us) write_op.cc:276] Timestamp: P: 1777902972853882 usec, L: 0
0504 13:56:34.448434 (+ 355us) write_op.cc:270] Start()
0504 13:56:34.448494 (+ 60us) write_op.cc:276] Timestamp: P: 1777902972872215 usec, L: 0
0504 13:56:34.449008 (+ 514us) write_op.cc:270] Start()
0504 13:56:34.449071 (+ 63us) write_op.cc:276] Timestamp: P: 1777902972887427 usec, L: 0
0504 13:56:34.449719 (+ 648us) write_op.cc:270] Start()
0504 13:56:34.449785 (+ 66us) write_op.cc:276] Timestamp: P: 1777902972916245 usec, L: 0
0504 13:56:34.450130 (+ 345us) write_op.cc:270] Start()
0504 13:56:34.450186 (+ 56us) write_op.cc:276] Timestamp: P: 1777902972990826 usec, L: 0
0504 13:56:34.450510 (+ 324us) write_op.cc:270] Start()
0504 13:56:34.450565 (+ 55us) write_op.cc:276] Timestamp: P: 1777902972997200 usec, L: 0
0504 13:56:34.450934 (+ 369us) write_op.cc:270] Start()
0504 13:56:34.450993 (+ 59us) write_op.cc:276] Timestamp: P: 1777902973021655 usec, L: 0
0504 13:56:34.451387 (+ 394us) write_op.cc:270] Start()
0504 13:56:34.451455 (+ 68us) write_op.cc:276] Timestamp: P: 1777902973037069 usec, L: 0
0504 13:56:34.451859 (+ 404us) write_op.cc:270] Start()
0504 13:56:34.451923 (+ 64us) write_op.cc:276] Timestamp: P: 1777902973097340 usec, L: 0
0504 13:56:34.452299 (+ 376us) write_op.cc:270] Start()
0504 13:56:34.452359 (+ 60us) write_op.cc:276] Timestamp: P: 1777902973100905 usec, L: 0
0504 13:56:34.452743 (+ 384us) write_op.cc:270] Start()
0504 13:56:34.452806 (+ 63us) write_op.cc:276] Timestamp: P: 1777902973121753 usec, L: 0
0504 13:56:34.453179 (+ 373us) write_op.cc:270] Start()
0504 13:56:34.453240 (+ 61us) write_op.cc:276] Timestamp: P: 1777902973156172 usec, L: 0
0504 13:56:34.453595 (+ 355us) write_op.cc:270] Start()
0504 13:56:34.453652 (+ 57us) write_op.cc:276] Timestamp: P: 1777902973220346 usec, L: 0
0504 13:56:34.454026 (+ 374us) write_op.cc:270] Start()
0504 13:56:34.454085 (+ 59us) write_op.cc:276] Timestamp: P: 1777902973476897 usec, L: 0
0504 13:56:34.454523 (+ 438us) write_op.cc:270] Start()
0504 13:56:34.454649 (+ 126us) write_op.cc:276] Timestamp: P: 1777902973523624 usec, L: 0
0504 13:56:34.455148 (+ 499us) write_op.cc:270] Start()
0504 13:56:34.455228 (+ 80us) write_op.cc:276] Timestamp: P: 1777902973586932 usec, L: 0
0504 13:56:34.455631 (+ 403us) write_op.cc:270] Start()
0504 13:56:34.455696 (+ 65us) write_op.cc:276] Timestamp: P: 1777902973596033 usec, L: 0
0504 13:56:34.456045 (+ 349us) write_op.cc:270] Start()
0504 13:56:34.456106 (+ 61us) write_op.cc:276] Timestamp: P: 1777902973608600 usec, L: 0
0504 13:56:34.456449 (+ 343us) write_op.cc:270] Start()
0504 13:56:34.456505 (+ 56us) write_op.cc:276] Timestamp: P: 1777902973665269 usec, L: 0
0504 13:56:34.456973 (+ 468us) write_op.cc:270] Start()
0504 13:56:34.457038 (+ 65us) write_op.cc:276] Timestamp: P: 1777902973680986 usec, L: 0
0504 13:56:34.457393 (+ 355us) write_op.cc:270] Start()
0504 13:56:34.457453 (+ 60us) write_op.cc:276] Timestamp: P: 1777902973690886 usec, L: 0
0504 13:56:34.457797 (+ 344us) write_op.cc:270] Start()
0504 13:56:34.457864 (+ 67us) write_op.cc:276] Timestamp: P: 1777902973726600 usec, L: 0
0504 13:56:34.458204 (+ 340us) write_op.cc:270] Start()
0504 13:56:34.458258 (+ 54us) write_op.cc:276] Timestamp: P: 1777902973763097 usec, L: 0
0504 13:56:34.458589 (+ 331us) write_op.cc:270] Start()
0504 13:56:34.458651 (+ 62us) write_op.cc:276] Timestamp: P: 1777902973772022 usec, L: 0
0504 13:56:34.460288 (+ 1637us) write_op.cc:270] Start()
0504 13:56:34.460349 (+ 61us) write_op.cc:276] Timestamp: P: 1777902973774846 usec, L: 0
0504 13:56:34.460702 (+ 353us) write_op.cc:270] Start()
0504 13:56:34.460760 (+ 58us) write_op.cc:276] Timestamp: P: 1777902973807936 usec, L: 0
0504 13:56:34.461126 (+ 366us) write_op.cc:270] Start()
0504 13:56:34.461191 (+ 65us) write_op.cc:276] Timestamp: P: 1777902973814673 usec, L: 0
0504 13:56:34.461550 (+ 359us) write_op.cc:270] Start()
0504 13:56:34.461617 (+ 67us) write_op.cc:276] Timestamp: P: 1777902973890387 usec, L: 0
0504 13:56:34.461996 (+ 379us) write_op.cc:270] Start()
0504 13:56:34.462055 (+ 59us) write_op.cc:276] Timestamp: P: 1777902973928445 usec, L: 0
0504 13:56:34.462448 (+ 393us) write_op.cc:270] Start()
0504 13:56:34.462573 (+ 125us) write_op.cc:276] Timestamp: P: 1777902974004994 usec, L: 0
0504 13:56:34.462939 (+ 366us) write_op.cc:270] Start()
0504 13:56:34.463002 (+ 63us) write_op.cc:276] Timestamp: P: 1777902974087776 usec, L: 0
0504 13:56:34.463392 (+ 390us) write_op.cc:270] Start()
0504 13:56:34.463455 (+ 63us) write_op.cc:276] Timestamp: P: 1777902974089518 usec, L: 0
0504 13:56:34.463840 (+ 385us) write_op.cc:270] Start()
0504 13:56:34.463898 (+ 58us) write_op.cc:276] Timestamp: P: 1777902974141586 usec, L: 0
0504 13:56:34.464265 (+ 367us) write_op.cc:270] Start()
0504 13:56:34.464325 (+ 60us) write_op.cc:276] Timestamp: P: 1777902974306081 usec, L: 0
0504 13:56:34.464697 (+ 372us) write_op.cc:270] Start()
0504 13:56:34.464764 (+ 67us) write_op.cc:276] Timestamp: P: 1777902974367504 usec, L: 0
0504 13:56:34.465188 (+ 424us) write_op.cc:270] Start()
0504 13:56:34.465251 (+ 63us) write_op.cc:276] Timestamp: P: 1777902974593428 usec, L: 0
0504 13:56:34.465619 (+ 368us) write_op.cc:270] Start()
0504 13:56:34.465678 (+ 59us) write_op.cc:276] Timestamp: P: 1777902974596292 usec, L: 0
0504 13:56:34.466067 (+ 389us) write_op.cc:270] Start()
0504 13:56:34.466133 (+ 66us) write_op.cc:276] Timestamp: P: 1777902974600192 usec, L: 0
0504 13:56:34.466517 (+ 384us) write_op.cc:270] Start()
0504 13:56:34.466601 (+ 84us) write_op.cc:276] Timestamp: P: 1777902974606139 usec, L: 0
0504 13:56:34.467034 (+ 433us) write_op.cc:270] Start()
0504 13:56:34.467109 (+ 75us) write_op.cc:276] Timestamp: P: 1777902974644749 usec, L: 0
0504 13:56:34.467524 (+ 415us) write_op.cc:270] Start()
0504 13:56:34.467587 (+ 63us) write_op.cc:276] Timestamp: P: 1777902974928382 usec, L: 0
0504 13:56:34.467972 (+ 385us) write_op.cc:270] Start()
0504 13:56:34.468033 (+ 61us) write_op.cc:276] Timestamp: P: 1777902974930195 usec, L: 0
0504 13:56:34.468388 (+ 355us) write_op.cc:270] Start()
0504 13:56:34.468462 (+ 74us) write_op.cc:276] Timestamp: P: 1777902975179833 usec, L: 0
0504 13:56:34.468852 (+ 390us) write_op.cc:270] Start()
0504 13:56:34.468911 (+ 59us) write_op.cc:276] Timestamp: P: 1777902975182011 usec, L: 0
0504 13:56:34.469269 (+ 358us) write_op.cc:270] Start()
0504 13:56:34.469328 (+ 59us) write_op.cc:276] Timestamp: P: 1777902975330215 usec, L: 0
0504 13:56:34.469742 (+ 414us) write_op.cc:270] Start()
0504 13:56:34.469806 (+ 64us) write_op.cc:276] Timestamp: P: 1777902975332773 usec, L: 0
0504 13:56:34.471104 (+ 1298us) write_op.cc:270] Start()
0504 13:56:34.471178 (+ 74us) write_op.cc:276] Timestamp: P: 1777902975356023 usec, L: 0
0504 13:56:34.471661 (+ 483us) write_op.cc:270] Start()
0504 13:56:34.471727 (+ 66us) write_op.cc:276] Timestamp: P: 1777902975380940 usec, L: 0
0504 13:56:34.472175 (+ 448us) write_op.cc:270] Start()
0504 13:56:34.472238 (+ 63us) write_op.cc:276] Timestamp: P: 1777902975394782 usec, L: 0
0504 13:56:34.472593 (+ 355us) write_op.cc:270] Start()
0504 13:56:34.472653 (+ 60us) write_op.cc:276] Timestamp: P: 1777902975404907 usec, L: 0
0504 13:56:34.473092 (+ 439us) write_op.cc:270] Start()
0504 13:56:34.473167 (+ 75us) write_op.cc:276] Timestamp: P: 1777902975764112 usec, L: 0
0504 13:56:34.473602 (+ 435us) write_op.cc:270] Start()
0504 13:56:34.473684 (+ 82us) write_op.cc:276] Timestamp: P: 1777902975768676 usec, L: 0
0504 13:56:34.474532 (+ 848us) write_op.cc:270] Start()
0504 13:56:34.474603 (+ 71us) write_op.cc:276] Timestamp: P: 1777902975961702 usec, L: 0
0504 13:56:34.475020 (+ 417us) write_op.cc:270] Start()
0504 13:56:34.475085 (+ 65us) write_op.cc:276] Timestamp: P: 1777902975985390 usec, L: 0
0504 13:56:34.475488 (+ 403us) write_op.cc:270] Start()
0504 13:56:34.475553 (+ 65us) write_op.cc:276] Timestamp: P: 1777902976101147 usec, L: 0
0504 13:56:34.475979 (+ 426us) write_op.cc:270] Start()
0504 13:56:34.476039 (+ 60us) write_op.cc:276] Timestamp: P: 1777902976156493 usec, L: 0
0504 13:56:34.476427 (+ 388us) write_op.cc:270] Start()
0504 13:56:34.476490 (+ 63us) write_op.cc:276] Timestamp: P: 1777902976183531 usec, L: 0
0504 13:56:34.476913 (+ 423us) write_op.cc:270] Start()
0504 13:56:34.476976 (+ 63us) write_op.cc:276] Timestamp: P: 1777902976229365 usec, L: 0
0504 13:56:34.477362 (+ 386us) write_op.cc:270] Start()
0504 13:56:34.477431 (+ 69us) write_op.cc:276] Timestamp: P: 1777902976254271 usec, L: 0
0504 13:56:34.477831 (+ 400us) write_op.cc:270] Start()
0504 13:56:34.477892 (+ 61us) write_op.cc:276] Timestamp: P: 1777902976341840 usec, L: 0
0504 13:56:34.478267 (+ 375us) write_op.cc:270] Start()
0504 13:56:34.478326 (+ 59us) write_op.cc:276] Timestamp: P: 1777902976372354 usec, L: 0
0504 13:56:34.478672 (+ 346us) write_op.cc:270] Start()
0504 13:56:34.478731 (+ 59us) write_op.cc:276] Timestamp: P: 1777902976383334 usec, L: 0
0504 13:56:34.479086 (+ 355us) write_op.cc:270] Start()
0504 13:56:34.479146 (+ 60us) write_op.cc:276] Timestamp: P: 1777902976423966 usec, L: 0
0504 13:56:34.479537 (+ 391us) write_op.cc:270] Start()
0504 13:56:34.479597 (+ 60us) write_op.cc:276] Timestamp: P: 1777902976454362 usec, L: 0
0504 13:56:34.495332 (+ 15735us) write_op.cc:270] Start()
0504 13:56:34.495415 (+ 83us) write_op.cc:276] Timestamp: P: 1777902976487883 usec, L: 0
0504 13:56:34.495890 (+ 475us) write_op.cc:270] Start()
0504 13:56:34.495960 (+ 70us) write_op.cc:276] Timestamp: P: 1777902976489655 usec, L: 0
0504 13:56:34.496365 (+ 405us) write_op.cc:270] Start()
0504 13:56:34.496439 (+ 74us) write_op.cc:276] Timestamp: P: 1777902976628962 usec, L: 0
0504 13:56:34.496840 (+ 401us) write_op.cc:270] Start()
0504 13:56:34.496904 (+ 64us) write_op.cc:276] Timestamp: P: 1777902976825644 usec, L: 0
0504 13:56:34.497277 (+ 373us) write_op.cc:270] Start()
0504 13:56:34.497339 (+ 62us) write_op.cc:276] Timestamp: P: 1777902976870347 usec, L: 0
0504 13:56:34.497707 (+ 368us) write_op.cc:270] Start()
0504 13:56:34.497770 (+ 63us) write_op.cc:276] Timestamp: P: 1777902976913908 usec, L: 0
0504 13:56:34.511125 (+ 13355us) write_op.cc:270] Start()
0504 13:56:34.511218 (+ 93us) write_op.cc:276] Timestamp: P: 1777902976942681 usec, L: 0
0504 13:56:34.511627 (+ 409us) write_op.cc:270] Start()
0504 13:56:34.511699 (+ 72us) write_op.cc:276] Timestamp: P: 1777902976974796 usec, L: 0
0504 13:56:34.512077 (+ 378us) write_op.cc:270] Start()
0504 13:56:34.512140 (+ 63us) write_op.cc:276] Timestamp: P: 1777902977009627 usec, L: 0
0504 13:56:34.512510 (+ 370us) write_op.cc:270] Start()
0504 13:56:34.512574 (+ 64us) write_op.cc:276] Timestamp: P: 1777902977203259 usec, L: 0
0504 13:56:34.512966 (+ 392us) write_op.cc:270] Start()
0504 13:56:34.513029 (+ 63us) write_op.cc:276] Timestamp: P: 1777902977311511 usec, L: 0
0504 13:56:34.513424 (+ 395us) write_op.cc:270] Start()
0504 13:56:34.513484 (+ 60us) write_op.cc:276] Timestamp: P: 1777902977314776 usec, L: 0
0504 13:56:34.513877 (+ 393us) write_op.cc:270] Start()
0504 13:56:34.513939 (+ 62us) write_op.cc:276] Timestamp: P: 1777902977498331 usec, L: 0
0504 13:56:34.514339 (+ 400us) write_op.cc:270] Start()
0504 13:56:34.514397 (+ 58us) write_op.cc:276] Timestamp: P: 1777902977512571 usec, L: 0
0504 13:56:34.514730 (+ 333us) write_op.cc:270] Start()
0504 13:56:34.514787 (+ 57us) write_op.cc:276] Timestamp: P: 1777902977557343 usec, L: 0
0504 13:56:34.515253 (+ 466us) write_op.cc:270] Start()
0504 13:56:34.515311 (+ 58us) write_op.cc:276] Timestamp: P: 1777902977562197 usec, L: 0
0504 13:56:34.544967 (+ 29656us) write_op.cc:270] Start()
0504 13:56:34.545064 (+ 97us) write_op.cc:276] Timestamp: P: 1777902977821626 usec, L: 0
0504 13:56:34.545545 (+ 481us) write_op.cc:270] St
I20260504 13:56:48.327566 9931 ts_tablet_manager.cc:1434] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.127s user 0.005s sys 0.002s
I20260504 13:56:48.328374 9931 tablet_bootstrap.cc:492] T 9f80ce19e8f9498b98952a371476f185 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:48.309496 10013 consensus_queue.cc:260] T cf11f17e95dc411294c755e17c57811e P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.341877 10013 ts_tablet_manager.cc:1434] T cf11f17e95dc411294c755e17c57811e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.057s user 0.005s sys 0.000s
I20260504 13:56:48.356288 9188 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]). Expected version 1 got 0
I20260504 13:56:48.358131 9188 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]). Expected version 1 got 0
I20260504 13:56:48.359762 9188 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 3 got 0
I20260504 13:56:48.367529 11480 consensus_queue.cc:1048] T 6dd139c347f7406aac340cc6ac994273 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:48.384397 11692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:48.385221 11692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: This master's current role is: LEADER
I20260504 13:56:48.395424 9931 tablet_bootstrap.cc:654] T 9f80ce19e8f9498b98952a371476f185 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:48.401733 10013 tablet_bootstrap.cc:492] T eef975650610423c88ce8e4cfb0b013b P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:48.412763 10013 tablet_bootstrap.cc:654] T eef975650610423c88ce8e4cfb0b013b P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:48.413883 9675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6323e1c10e7643f8a28f04e285f620e9" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:48.414530 9675 raft_consensus.cc:2468] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:48.417735 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6323e1c10e7643f8a28f04e285f620e9" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:48.418265 9539 raft_consensus.cc:2468] T 6323e1c10e7643f8a28f04e285f620e9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:48.417907 9768 leader_election.cc:304] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:48.424836 11480 consensus_queue.cc:1048] T 644170f17f1b45a19677d958b9745e16 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:48.437079 11476 raft_consensus.cc:2804] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:48.437400 11476 raft_consensus.cc:493] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:48.437669 11476 raft_consensus.cc:3060] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:48.444242 11476 raft_consensus.cc:515] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:48.452077 11476 leader_election.cc:290] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:48.456017 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6323e1c10e7643f8a28f04e285f620e9" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:48.456104 9675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6323e1c10e7643f8a28f04e285f620e9" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:48.456560 9533 raft_consensus.cc:3060] T 6323e1c10e7643f8a28f04e285f620e9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:48.456578 9675 raft_consensus.cc:3060] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:48.467392 11641 consensus_queue.cc:1048] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:48.469779 9533 raft_consensus.cc:2468] T 6323e1c10e7643f8a28f04e285f620e9 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:48.468497 9931 tablet_bootstrap.cc:492] T 9f80ce19e8f9498b98952a371476f185 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:48.491823 9931 ts_tablet_manager.cc:1403] T 9f80ce19e8f9498b98952a371476f185 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.164s user 0.006s sys 0.005s
I20260504 13:56:48.511828 10013 tablet_bootstrap.cc:492] T eef975650610423c88ce8e4cfb0b013b P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:48.512472 10013 ts_tablet_manager.cc:1403] T eef975650610423c88ce8e4cfb0b013b P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.111s user 0.010s sys 0.002s
I20260504 13:56:48.515532 10013 raft_consensus.cc:359] T eef975650610423c88ce8e4cfb0b013b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.526237 9675 raft_consensus.cc:2468] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:48.494863 9931 raft_consensus.cc:359] T 9f80ce19e8f9498b98952a371476f185 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.531997 9931 raft_consensus.cc:385] T 9f80ce19e8f9498b98952a371476f185 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:48.532313 9931 raft_consensus.cc:740] T 9f80ce19e8f9498b98952a371476f185 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:48.532948 9931 consensus_queue.cc:260] T 9f80ce19e8f9498b98952a371476f185 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.547441 9931 ts_tablet_manager.cc:1434] T 9f80ce19e8f9498b98952a371476f185 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.055s user 0.003s sys 0.002s
I20260504 13:56:48.548677 9931 tablet_bootstrap.cc:492] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:48.559298 10013 raft_consensus.cc:385] T eef975650610423c88ce8e4cfb0b013b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:48.559578 10013 raft_consensus.cc:740] T eef975650610423c88ce8e4cfb0b013b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:48.560133 10013 consensus_queue.cc:260] T eef975650610423c88ce8e4cfb0b013b P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.562453 10013 ts_tablet_manager.cc:1434] T eef975650610423c88ce8e4cfb0b013b P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s
I20260504 13:56:48.574183 10013 tablet_bootstrap.cc:492] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:48.577893 9588 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:48.578276 11701 raft_consensus.cc:493] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:48.578685 11701 raft_consensus.cc:515] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.580523 11701 leader_election.cc:290] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:48.583148 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7989338bc4bf4bad8fb8fe37587d0a16" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:48.585624 9618 leader_election.cc:343] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:48.615446 10013 tablet_bootstrap.cc:654] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:48.655449 10013 tablet_bootstrap.cc:492] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:48.655864 10013 ts_tablet_manager.cc:1403] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.082s user 0.005s sys 0.005s
I20260504 13:56:48.658156 10013 raft_consensus.cc:359] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.658748 10013 raft_consensus.cc:385] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:48.472092 9767 leader_election.cc:304] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888; no voters:
W20260504 13:56:48.661962 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 189791us
I20260504 13:56:48.658986 10013 raft_consensus.cc:740] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:48.679605 10013 consensus_queue.cc:260] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.680387 11701 raft_consensus.cc:493] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:48.680811 11701 raft_consensus.cc:515] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.682646 11701 leader_election.cc:290] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:48.685489 9821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9e66dacf5794b6195575556be889bce" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:48.691624 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9e66dacf5794b6195575556be889bce" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:48.693813 9618 leader_election.cc:343] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:48.696600 9620 leader_election.cc:343] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:48.697044 9620 leader_election.cc:304] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:48.699618 11701 raft_consensus.cc:2749] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:48.695869 10013 ts_tablet_manager.cc:1434] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s
I20260504 13:56:48.700778 10013 tablet_bootstrap.cc:492] T 37b1664312d347288680d97b62c2aecb P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:48.720014 11641 raft_consensus.cc:493] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:48.720453 11641 raft_consensus.cc:515] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.722396 11641 leader_election.cc:290] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:48.724135 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ff0d52a90c4056880190dbd936cb8e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:48.726413 9618 leader_election.cc:343] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:48.727629 9821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ff0d52a90c4056880190dbd936cb8e" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:48.728842 9620 leader_election.cc:343] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:48.729266 9620 leader_election.cc:304] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:48.730145 11641 raft_consensus.cc:2749] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:48.750176 9830 raft_consensus.cc:3060] T 9f80ce19e8f9498b98952a371476f185 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:48.754235 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7989338bc4bf4bad8fb8fe37587d0a16" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:48.761161 11706 raft_consensus.cc:2804] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:48.764576 10013 tablet_bootstrap.cc:654] T 37b1664312d347288680d97b62c2aecb P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:48.768693 9620 leader_election.cc:343] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:48.769049 9620 leader_election.cc:304] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:48.770224 11641 raft_consensus.cc:2749] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:48.788048 11480 raft_consensus.cc:493] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:48.788442 11480 raft_consensus.cc:515] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.792496 11701 consensus_queue.cc:1048] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:48.791112 11480 leader_election.cc:290] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:48.795126 9675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e29a7365caf04506b15089f10e6c1677" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:48.795771 9675 raft_consensus.cc:2468] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:48.797657 9473 leader_election.cc:304] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:48.798442 11669 raft_consensus.cc:2804] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:48.798779 11669 raft_consensus.cc:493] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:48.799070 11669 raft_consensus.cc:3060] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:48.808430 11669 raft_consensus.cc:515] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.809275 10013 tablet_bootstrap.cc:492] T 37b1664312d347288680d97b62c2aecb P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:48.809734 10013 ts_tablet_manager.cc:1403] T 37b1664312d347288680d97b62c2aecb P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.109s user 0.007s sys 0.001s
I20260504 13:56:48.840092 9931 tablet_bootstrap.cc:700] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888: Time spent opening tablet: real 0.286s user 0.003s sys 0.000s
I20260504 13:56:48.841126 9931 tablet_bootstrap.cc:654] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:48.844135 11669 leader_election.cc:290] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:48.845510 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e29a7365caf04506b15089f10e6c1677" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:48.846048 9672 raft_consensus.cc:3060] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:48.856289 9672 raft_consensus.cc:2468] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:48.857077 9473 leader_election.cc:304] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:48.857985 11669 raft_consensus.cc:2804] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:48.832492 10013 raft_consensus.cc:359] T 37b1664312d347288680d97b62c2aecb P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.859678 10013 raft_consensus.cc:385] T 37b1664312d347288680d97b62c2aecb P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:48.859989 10013 raft_consensus.cc:740] T 37b1664312d347288680d97b62c2aecb P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:48.867519 10013 consensus_queue.cc:260] T 37b1664312d347288680d97b62c2aecb P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.875258 11706 raft_consensus.cc:697] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:48.876348 11706 consensus_queue.cc:237] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } }
I20260504 13:56:48.881037 9931 tablet_bootstrap.cc:492] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:48.881620 9931 ts_tablet_manager.cc:1403] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.333s user 0.011s sys 0.000s
I20260504 13:56:48.887463 11711 raft_consensus.cc:493] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:48.888160 11711 raft_consensus.cc:515] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.889204 9931 raft_consensus.cc:359] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.889905 9931 raft_consensus.cc:385] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:48.890192 9931 raft_consensus.cc:740] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:48.890486 11711 leader_election.cc:290] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:48.891489 11669 raft_consensus.cc:697] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:48.892336 11669 consensus_queue.cc:237] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.892807 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c09e9076214405eaa8846fe08158bb3" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:48.893344 9683 raft_consensus.cc:2468] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:48.893823 11661 raft_consensus.cc:493] T 8d6db45d2e5c41a3b30cede3301694d3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:48.894778 10013 ts_tablet_manager.cc:1434] T 37b1664312d347288680d97b62c2aecb P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.085s user 0.006s sys 0.001s
I20260504 13:56:48.895123 11661 raft_consensus.cc:515] T 8d6db45d2e5c41a3b30cede3301694d3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.895792 10013 tablet_bootstrap.cc:492] T 2c09e9076214405eaa8846fe08158bb3 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:48.898923 11661 leader_election.cc:290] T 8d6db45d2e5c41a3b30cede3301694d3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:48.900071 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d6db45d2e5c41a3b30cede3301694d3" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:48.900702 9683 raft_consensus.cc:2393] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 1: Already voted for candidate d4051d2bf0bf410eb8acd107560df888 in this term.
I20260504 13:56:48.931531 9931 consensus_queue.cc:260] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.941809 9931 ts_tablet_manager.cc:1434] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.060s user 0.006s sys 0.002s
I20260504 13:56:48.943100 9931 tablet_bootstrap.cc:492] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:48.947326 9828 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e29a7365caf04506b15089f10e6c1677" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:48.953444 11641 consensus_queue.cc:1048] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:48.956367 11701 raft_consensus.cc:493] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:48.956791 11701 raft_consensus.cc:515] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:48.958566 11701 leader_election.cc:290] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:48.960031 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e29a7365caf04506b15089f10e6c1677" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:48.960135 9823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d6db45d2e5c41a3b30cede3301694d3" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:48.962904 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef975650610423c88ce8e4cfb0b013b" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
W20260504 13:56:48.964551 9620 leader_election.cc:343] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:48.965314 9533 raft_consensus.cc:3060] T 37b1664312d347288680d97b62c2aecb P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:48.967928 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef975650610423c88ce8e4cfb0b013b" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:48.968461 9530 raft_consensus.cc:2468] T eef975650610423c88ce8e4cfb0b013b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 0.
I20260504 13:56:48.969030 9931 tablet_bootstrap.cc:654] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:48.969612 9474 leader_election.cc:343] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:48.971906 9768 leader_election.cc:304] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:48.973806 11715 raft_consensus.cc:2804] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:48.974169 11715 raft_consensus.cc:493] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:48.974454 11715 raft_consensus.cc:3060] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:48.976435 9618 leader_election.cc:304] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:48.979849 11570 raft_consensus.cc:2804] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:48.979795 9474 leader_election.cc:304] T 8d6db45d2e5c41a3b30cede3301694d3 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:48.980155 11570 raft_consensus.cc:493] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:48.980443 11570 raft_consensus.cc:3060] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:48.981166 11433 raft_consensus.cc:3060] T 8d6db45d2e5c41a3b30cede3301694d3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:48.992440 11709 consensus_queue.cc:1048] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:49.002892 11433 raft_consensus.cc:2749] T 8d6db45d2e5c41a3b30cede3301694d3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260504 13:56:49.004496 9474 leader_election.cc:343] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:48.997141 11570 raft_consensus.cc:515] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.013278 11715 raft_consensus.cc:515] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.019814 11715 leader_election.cc:290] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:49.019954 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c09e9076214405eaa8846fe08158bb3" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:49.020424 9684 raft_consensus.cc:3060] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:49.023114 9834 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef975650610423c88ce8e4cfb0b013b" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:49.024022 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef975650610423c88ce8e4cfb0b013b" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:49.024497 9530 raft_consensus.cc:3060] T eef975650610423c88ce8e4cfb0b013b P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:49.030582 9530 raft_consensus.cc:2468] T eef975650610423c88ce8e4cfb0b013b P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beceb6317444247b31b44a6581707c6 in term 1.
I20260504 13:56:49.031981 9684 raft_consensus.cc:2468] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
W20260504 13:56:49.032243 9620 leader_election.cc:343] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:49.036631 9768 leader_election.cc:304] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:49.036612 9618 leader_election.cc:304] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters: d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:49.037273 11641 raft_consensus.cc:2804] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:49.037436 11714 raft_consensus.cc:2804] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:49.037698 11641 raft_consensus.cc:697] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Running, Role: LEADER
I20260504 13:56:49.038110 11714 raft_consensus.cc:697] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:49.038465 11641 consensus_queue.cc:237] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.039379 9931 tablet_bootstrap.cc:492] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:49.039825 9931 ts_tablet_manager.cc:1403] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.097s user 0.008s sys 0.007s
I20260504 13:56:49.042383 9931 raft_consensus.cc:359] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.032161 10013 tablet_bootstrap.cc:654] T 2c09e9076214405eaa8846fe08158bb3 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.038890 11714 consensus_queue.cc:237] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.051085 9931 raft_consensus.cc:385] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.053939 9931 raft_consensus.cc:740] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.055506 9931 consensus_queue.cc:260] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.058704 9931 ts_tablet_manager.cc:1434] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.019s user 0.003s sys 0.002s
I20260504 13:56:49.064504 9931 tablet_bootstrap.cc:492] T e29a7365caf04506b15089f10e6c1677 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:49.072803 9931 tablet_bootstrap.cc:654] T e29a7365caf04506b15089f10e6c1677 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.035085 11570 leader_election.cc:290] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563)
I20260504 13:56:49.095901 9931 tablet_bootstrap.cc:492] T e29a7365caf04506b15089f10e6c1677 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:49.096352 9931 ts_tablet_manager.cc:1403] T e29a7365caf04506b15089f10e6c1677 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.009s
I20260504 13:56:49.100490 9931 raft_consensus.cc:359] T e29a7365caf04506b15089f10e6c1677 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.101080 9931 raft_consensus.cc:385] T e29a7365caf04506b15089f10e6c1677 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.101323 9931 raft_consensus.cc:740] T e29a7365caf04506b15089f10e6c1677 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.102029 9931 consensus_queue.cc:260] T e29a7365caf04506b15089f10e6c1677 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.104558 9931 ts_tablet_manager.cc:1434] T e29a7365caf04506b15089f10e6c1677 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20260504 13:56:49.105652 9931 tablet_bootstrap.cc:492] T eef975650610423c88ce8e4cfb0b013b P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:49.112221 11570 consensus_queue.cc:1048] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:49.146551 11641 raft_consensus.cc:493] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:49.147042 11641 raft_consensus.cc:515] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.148877 11641 leader_election.cc:290] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117)
I20260504 13:56:49.150187 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c09e9076214405eaa8846fe08158bb3" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:49.156090 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b942037c0e45a6b3cccba288b90b00" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:49.156270 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b942037c0e45a6b3cccba288b90b00" candidate_uuid: "5beceb6317444247b31b44a6581707c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
W20260504 13:56:49.157470 9620 leader_election.cc:343] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:49.159039 9618 leader_election.cc:343] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:49.159406 9618 leader_election.cc:304] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5beceb6317444247b31b44a6581707c6; no voters: 7b79fe7dd8934dd996393a2d96eabf00, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:49.160437 11701 raft_consensus.cc:2749] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:49.169101 11706 raft_consensus.cc:493] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:49.172480 11706 raft_consensus.cc:515] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.175611 11706 leader_election.cc:290] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:49.176441 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf11f17e95dc411294c755e17c57811e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:49.176960 9684 raft_consensus.cc:2468] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:49.159147 9931 tablet_bootstrap.cc:654] T eef975650610423c88ce8e4cfb0b013b P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.178026 9768 leader_election.cc:304] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:49.178885 11706 raft_consensus.cc:2804] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:49.179317 11706 raft_consensus.cc:493] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:49.179715 11706 raft_consensus.cc:3060] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:49.193277 11714 consensus_queue.cc:1048] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:49.201153 10013 tablet_bootstrap.cc:492] T 2c09e9076214405eaa8846fe08158bb3 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:49.201656 10013 ts_tablet_manager.cc:1403] T 2c09e9076214405eaa8846fe08158bb3 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.306s user 0.007s sys 0.005s
I20260504 13:56:49.203629 9931 tablet_bootstrap.cc:492] T eef975650610423c88ce8e4cfb0b013b P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:49.204085 9931 ts_tablet_manager.cc:1403] T eef975650610423c88ce8e4cfb0b013b P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.099s user 0.005s sys 0.005s
I20260504 13:56:49.207278 9931 raft_consensus.cc:359] T eef975650610423c88ce8e4cfb0b013b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.211350 9931 raft_consensus.cc:385] T eef975650610423c88ce8e4cfb0b013b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.212126 9931 raft_consensus.cc:740] T eef975650610423c88ce8e4cfb0b013b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.213138 11734 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 806407748d564439807fe5d73db9abda)
I20260504 13:56:49.213788 11734 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:49.213714 9931 consensus_queue.cc:260] T eef975650610423c88ce8e4cfb0b013b P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.211283 11706 raft_consensus.cc:515] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.216473 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 5 candidate_status { last_received { term: 4 index: 371 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:49.219211 9352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 5 candidate_status { last_received { term: 4 index: 371 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:49.219971 10013 raft_consensus.cc:359] T 2c09e9076214405eaa8846fe08158bb3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.220245 9352 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6caedf407d13459e8fd477e8b60b7dc7 in term 4.
I20260504 13:56:49.220549 10013 raft_consensus.cc:385] T 2c09e9076214405eaa8846fe08158bb3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.220806 10013 raft_consensus.cc:740] T 2c09e9076214405eaa8846fe08158bb3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.223255 10690 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7, ec9f4c3a43d84fc9a420dd10723c01a1; no voters: 806407748d564439807fe5d73db9abda
I20260504 13:56:49.225353 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c09e9076214405eaa8846fe08158bb3" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:49.229226 9538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf11f17e95dc411294c755e17c57811e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:49.229854 9538 raft_consensus.cc:2468] T cf11f17e95dc411294c755e17c57811e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:49.277575 11730 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 806407748d564439807fe5d73db9abda)
I20260504 13:56:49.278131 11730 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:49.280565 11730 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:56:49.280907 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 4 index: 371 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:49.281205 10013 consensus_queue.cc:260] T 2c09e9076214405eaa8846fe08158bb3 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.282047 9533 raft_consensus.cc:3060] T 2c09e9076214405eaa8846fe08158bb3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:49.283555 11734 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:49.284046 11734 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260504 13:56:49.284335 11734 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 806407748d564439807fe5d73db9abda)
I20260504 13:56:49.218015 9931 ts_tablet_manager.cc:1434] T eef975650610423c88ce8e4cfb0b013b P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.013s user 0.004s sys 0.002s
W20260504 13:56:49.252266 9767 leader_election.cc:343] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:49.285569 9931 tablet_bootstrap.cc:492] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:49.289211 9533 raft_consensus.cc:2468] T 2c09e9076214405eaa8846fe08158bb3 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:49.291488 9931 tablet_bootstrap.cc:654] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.294684 10013 ts_tablet_manager.cc:1434] T 2c09e9076214405eaa8846fe08158bb3 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.093s user 0.005s sys 0.001s
I20260504 13:56:49.295621 10013 tablet_bootstrap.cc:492] T 6f4c8ffb2175493a981c18676485ac04 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:49.303282 10013 tablet_bootstrap.cc:654] T 6f4c8ffb2175493a981c18676485ac04 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.308478 9931 tablet_bootstrap.cc:492] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:49.308856 9931 ts_tablet_manager.cc:1403] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.004s
I20260504 13:56:49.311070 9931 raft_consensus.cc:359] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.311681 9931 raft_consensus.cc:385] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.311985 9931 raft_consensus.cc:740] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.312568 9931 consensus_queue.cc:260] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.314425 9931 ts_tablet_manager.cc:1434] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260504 13:56:49.314357 9675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf11f17e95dc411294c755e17c57811e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:49.314978 9675 raft_consensus.cc:3060] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:49.320307 9931 tablet_bootstrap.cc:492] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:49.320676 11734 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 4 FOLLOWER]: Advancing to term 5
I20260504 13:56:49.319595 9182 catalog_manager.cc:5671] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:49.321492 9182 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 3 got 0
I20260504 13:56:49.323494 9182 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 88619eaa917d4c33a910fc1347ab126e (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]). Expected version 1 got 0
I20260504 13:56:49.324751 10013 tablet_bootstrap.cc:492] T 6f4c8ffb2175493a981c18676485ac04 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:49.325136 10013 ts_tablet_manager.cc:1403] T 6f4c8ffb2175493a981c18676485ac04 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.016s
I20260504 13:56:49.324883 11709 consensus_queue.cc:1048] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:49.326462 9182 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4]). Expected version 1 got 0
I20260504 13:56:49.328469 9182 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
I20260504 13:56:49.329007 9931 tablet_bootstrap.cc:654] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.332918 10013 raft_consensus.cc:359] T 6f4c8ffb2175493a981c18676485ac04 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.333472 10013 raft_consensus.cc:385] T 6f4c8ffb2175493a981c18676485ac04 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.333709 10013 raft_consensus.cc:740] T 6f4c8ffb2175493a981c18676485ac04 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.334208 10013 consensus_queue.cc:260] T 6f4c8ffb2175493a981c18676485ac04 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.335588 10013 ts_tablet_manager.cc:1434] T 6f4c8ffb2175493a981c18676485ac04 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260504 13:56:49.336861 10013 tablet_bootstrap.cc:492] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:49.338405 9182 catalog_manager.cc:5671] T 8c13c4cd2f66426f8e7e81eeac7bc0f3 P 7b79fe7dd8934dd996393a2d96eabf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } }
I20260504 13:56:49.341957 11706 leader_election.cc:290] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:49.345067 9182 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 2ddf890c17964e7f90409ebe3c4034ef (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]). Expected version 2 got 0
I20260504 13:56:49.345500 10013 tablet_bootstrap.cc:654] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.345865 9182 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]). Expected version 1 got 0
I20260504 13:56:49.350205 9182 catalog_manager.cc:5763] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117) does not have the latest schema for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]). Expected version 1 got 0
I20260504 13:56:49.372972 11706 consensus_queue.cc:1048] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260504 13:56:49.377287 10690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.252:44781, user_credentials={real_user=slave}} blocked reactor thread for 154186us
I20260504 13:56:49.378654 10814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 4 index: 371 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:49.380491 9328 leader_election.cc:304] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec9f4c3a43d84fc9a420dd10723c01a1; no voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda
I20260504 13:56:49.381384 11730 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260504 13:56:49.423465 10013 tablet_bootstrap.cc:492] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:49.423877 10013 ts_tablet_manager.cc:1403] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.087s user 0.009s sys 0.020s
I20260504 13:56:49.435837 10013 raft_consensus.cc:359] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.436455 10013 raft_consensus.cc:385] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.436704 10013 raft_consensus.cc:740] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.437186 11734 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:49.438174 9675 raft_consensus.cc:2468] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:49.439707 9768 leader_election.cc:304] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:49.440471 11711 raft_consensus.cc:2804] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:49.440987 11711 raft_consensus.cc:697] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:49.442001 11711 consensus_queue.cc:237] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.437321 10013 consensus_queue.cc:260] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.453986 10013 ts_tablet_manager.cc:1434] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260504 13:56:49.456162 9931 tablet_bootstrap.cc:492] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:49.456549 9931 ts_tablet_manager.cc:1403] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.141s user 0.030s sys 0.008s
I20260504 13:56:49.461711 9931 raft_consensus.cc:359] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.462296 9931 raft_consensus.cc:385] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.462534 9931 raft_consensus.cc:740] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.463164 9931 consensus_queue.cc:260] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.464650 9931 ts_tablet_manager.cc:1434] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.008s user 0.003s sys 0.001s
I20260504 13:56:49.465374 10013 tablet_bootstrap.cc:492] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:49.468254 11709 raft_consensus.cc:493] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:49.468746 11709 raft_consensus.cc:515] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.471311 10013 tablet_bootstrap.cc:654] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.471407 11709 leader_election.cc:290] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:49.471501 9674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f4c8ffb2175493a981c18676485ac04" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:49.472106 9674 raft_consensus.cc:2468] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:49.473261 9768 leader_election.cc:304] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:49.473973 11709 raft_consensus.cc:2804] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:49.474570 11709 raft_consensus.cc:493] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:49.475033 11709 raft_consensus.cc:3060] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:49.479486 11706 raft_consensus.cc:493] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:49.479948 11706 raft_consensus.cc:515] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.481761 10013 tablet_bootstrap.cc:492] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:49.482132 10013 ts_tablet_manager.cc:1403] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.012s
I20260504 13:56:49.482216 11706 leader_election.cc:290] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:49.482755 9674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7989338bc4bf4bad8fb8fe37587d0a16" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:49.483369 9674 raft_consensus.cc:2468] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:49.484529 9768 leader_election.cc:304] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:49.484498 10013 raft_consensus.cc:359] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.485443 11604 raft_consensus.cc:2804] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:49.485754 11604 raft_consensus.cc:493] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:49.486085 11604 raft_consensus.cc:3060] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:49.493069 11604 raft_consensus.cc:515] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.495764 11604 leader_election.cc:290] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:49.485082 10013 raft_consensus.cc:385] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.496294 10013 raft_consensus.cc:740] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.497648 9675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7989338bc4bf4bad8fb8fe37587d0a16" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:49.497761 10013 consensus_queue.cc:260] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.498202 9675 raft_consensus.cc:3060] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:49.499936 10013 ts_tablet_manager.cc:1434] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20260504 13:56:49.503731 11709 raft_consensus.cc:515] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.505973 10814 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 5 FOLLOWER]: Rejecting Update request from peer 806407748d564439807fe5d73db9abda for earlier term 4. Current term is 5. Ops: [4.372-4.372]
I20260504 13:56:49.506242 11734 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 5 election: Requested vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:49.507578 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 5 candidate_status { last_received { term: 4 index: 371 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda"
I20260504 13:56:49.507820 9352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 5 candidate_status { last_received { term: 4 index: 371 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1"
I20260504 13:56:49.509941 9931 tablet_bootstrap.cc:492] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:49.514564 10690 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:49.515460 9931 tablet_bootstrap.cc:654] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.524107 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf11f17e95dc411294c755e17c57811e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:49.524575 9533 raft_consensus.cc:3060] T cf11f17e95dc411294c755e17c57811e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:49.529892 9540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7989338bc4bf4bad8fb8fe37587d0a16" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:49.531750 11624 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 }, Status: INVALID_TERM, Last received: 4.371, Next index: 372, Last known committed idx: 371, Time since last communication: 0.000s
I20260504 13:56:49.531984 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f4c8ffb2175493a981c18676485ac04" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:49.532518 9535 raft_consensus.cc:2468] T 6f4c8ffb2175493a981c18676485ac04 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:49.532655 11624 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 4 LEADER]: Stepping down as leader of term 4
I20260504 13:56:49.532963 11624 raft_consensus.cc:740] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 806407748d564439807fe5d73db9abda, State: Running, Role: LEADER
I20260504 13:56:49.533123 11709 leader_election.cc:290] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:49.533196 9674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f4c8ffb2175493a981c18676485ac04" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:49.533792 9674 raft_consensus.cc:3060] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:49.533727 11624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 4.372, Last appended by leader: 372, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:49.535524 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7989338bc4bf4bad8fb8fe37587d0a16" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:49.536542 11624 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 4 FOLLOWER]: Advancing to term 5
I20260504 13:56:49.545398 9674 raft_consensus.cc:2468] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:49.554754 9768 leader_election.cc:304] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:49.555580 11711 raft_consensus.cc:493] T e29a7365caf04506b15089f10e6c1677 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:49.555974 11711 raft_consensus.cc:515] T e29a7365caf04506b15089f10e6c1677 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.556548 11604 raft_consensus.cc:2804] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:49.556977 11604 raft_consensus.cc:697] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:49.557785 11711 leader_election.cc:290] T e29a7365caf04506b15089f10e6c1677 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:49.557634 11604 consensus_queue.cc:237] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.559768 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e29a7365caf04506b15089f10e6c1677" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:49.560482 9672 raft_consensus.cc:2393] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:49.562156 9533 raft_consensus.cc:2468] T cf11f17e95dc411294c755e17c57811e P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:49.568518 9675 raft_consensus.cc:2468] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:49.569322 10013 tablet_bootstrap.cc:492] T 7989338bc4bf4bad8fb8fe37587d0a16 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:49.569752 11734 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260504 13:56:49.573789 9768 leader_election.cc:304] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:49.574440 11706 raft_consensus.cc:2804] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:49.575088 10013 tablet_bootstrap.cc:654] T 7989338bc4bf4bad8fb8fe37587d0a16 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.575127 11706 raft_consensus.cc:697] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:49.575935 11706 consensus_queue.cc:237] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:49.588127 9767 leader_election.cc:343] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
W20260504 13:56:49.589464 9767 leader_election.cc:343] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:49.590356 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e29a7365caf04506b15089f10e6c1677" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:49.594877 10013 tablet_bootstrap.cc:492] T 7989338bc4bf4bad8fb8fe37587d0a16 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:49.595271 10013 ts_tablet_manager.cc:1403] T 7989338bc4bf4bad8fb8fe37587d0a16 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.003s
I20260504 13:56:49.597460 10013 raft_consensus.cc:359] T 7989338bc4bf4bad8fb8fe37587d0a16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.604789 9767 leader_election.cc:304] T e29a7365caf04506b15089f10e6c1677 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:49.605548 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f4c8ffb2175493a981c18676485ac04" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:49.605950 11711 raft_consensus.cc:3060] T e29a7365caf04506b15089f10e6c1677 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:49.606429 9533 raft_consensus.cc:3060] T 6f4c8ffb2175493a981c18676485ac04 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:49.610975 9931 tablet_bootstrap.cc:492] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:49.611413 9931 ts_tablet_manager.cc:1403] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.102s user 0.014s sys 0.006s
I20260504 13:56:49.614342 11711 raft_consensus.cc:2749] T e29a7365caf04506b15089f10e6c1677 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:49.615828 9736 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:49.625926 9931 raft_consensus.cc:359] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.598058 10013 raft_consensus.cc:385] T 7989338bc4bf4bad8fb8fe37587d0a16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.628845 10013 raft_consensus.cc:740] T 7989338bc4bf4bad8fb8fe37587d0a16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.642901 10013 consensus_queue.cc:260] T 7989338bc4bf4bad8fb8fe37587d0a16 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.644423 10013 ts_tablet_manager.cc:1434] T 7989338bc4bf4bad8fb8fe37587d0a16 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.049s user 0.004s sys 0.000s
I20260504 13:56:49.645573 9931 raft_consensus.cc:385] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.645836 9931 raft_consensus.cc:740] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.646438 9931 consensus_queue.cc:260] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.653024 9931 ts_tablet_manager.cc:1434] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.041s user 0.003s sys 0.002s
I20260504 13:56:49.653843 9931 tablet_bootstrap.cc:492] T 6677a93d01d54f15a506ae1474bcb501 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:49.656533 9533 raft_consensus.cc:2468] T 6f4c8ffb2175493a981c18676485ac04 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:49.663578 11480 raft_consensus.cc:493] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:49.663968 11480 raft_consensus.cc:515] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.665645 11480 leader_election.cc:290] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:49.667213 9675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6677a93d01d54f15a506ae1474bcb501" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:49.667773 9675 raft_consensus.cc:2468] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:49.667840 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6677a93d01d54f15a506ae1474bcb501" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:49.668915 9473 leader_election.cc:304] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:49.669812 11480 raft_consensus.cc:2804] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:49.670089 11480 raft_consensus.cc:493] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:49.670375 11480 raft_consensus.cc:3060] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:56:49.680181 9474 leader_election.cc:343] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:49.682042 11480 raft_consensus.cc:515] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.684401 9675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6677a93d01d54f15a506ae1474bcb501" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:49.684887 9675 raft_consensus.cc:3060] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:49.686508 9931 tablet_bootstrap.cc:654] T 6677a93d01d54f15a506ae1474bcb501 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.693230 9675 raft_consensus.cc:2468] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:49.694145 9473 leader_election.cc:304] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:49.695413 11433 raft_consensus.cc:2804] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:49.695668 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6677a93d01d54f15a506ae1474bcb501" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:49.695837 11433 raft_consensus.cc:697] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:49.696571 11433 consensus_queue.cc:237] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
W20260504 13:56:49.697942 9474 leader_election.cc:343] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:56:49.720134 11480 leader_election.cc:290] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:49.724892 9931 tablet_bootstrap.cc:492] T 6677a93d01d54f15a506ae1474bcb501 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:49.725404 9931 ts_tablet_manager.cc:1403] T 6677a93d01d54f15a506ae1474bcb501 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.004s
I20260504 13:56:49.725776 10013 tablet_bootstrap.cc:492] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:49.728714 9931 raft_consensus.cc:359] T 6677a93d01d54f15a506ae1474bcb501 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.729457 9931 raft_consensus.cc:385] T 6677a93d01d54f15a506ae1474bcb501 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.729768 9931 raft_consensus.cc:740] T 6677a93d01d54f15a506ae1474bcb501 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.730374 9931 consensus_queue.cc:260] T 6677a93d01d54f15a506ae1474bcb501 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.731055 10013 tablet_bootstrap.cc:654] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.733943 9931 ts_tablet_manager.cc:1434] T 6677a93d01d54f15a506ae1474bcb501 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260504 13:56:49.743338 9931 tablet_bootstrap.cc:492] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:49.749531 9931 tablet_bootstrap.cc:654] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.779982 10013 tablet_bootstrap.cc:492] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:49.780471 10013 ts_tablet_manager.cc:1403] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.000s
I20260504 13:56:49.783144 11433 consensus_queue.cc:1048] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:49.787133 11604 raft_consensus.cc:493] T eef975650610423c88ce8e4cfb0b013b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:49.787606 11604 raft_consensus.cc:515] T eef975650610423c88ce8e4cfb0b013b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.789135 9931 tablet_bootstrap.cc:492] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:49.789294 11604 leader_election.cc:290] T eef975650610423c88ce8e4cfb0b013b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:49.789634 9931 ts_tablet_manager.cc:1403] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.025s
I20260504 13:56:49.790493 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef975650610423c88ce8e4cfb0b013b" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:49.791914 9931 raft_consensus.cc:359] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.793291 9931 raft_consensus.cc:385] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.793586 9931 raft_consensus.cc:740] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.794203 9931 consensus_queue.cc:260] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.796958 10013 raft_consensus.cc:359] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.797565 10013 raft_consensus.cc:385] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.797792 10013 raft_consensus.cc:740] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.803418 10013 consensus_queue.cc:260] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.807801 10013 ts_tablet_manager.cc:1434] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20260504 13:56:49.796895 9931 ts_tablet_manager.cc:1434] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260504 13:56:49.808658 10013 tablet_bootstrap.cc:492] T e9e66dacf5794b6195575556be889bce P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:49.808774 9931 tablet_bootstrap.cc:492] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:49.809227 11570 consensus_queue.cc:1048] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:49.816253 9833 raft_consensus.cc:3060] T eef975650610423c88ce8e4cfb0b013b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:49.818750 9931 tablet_bootstrap.cc:654] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.825047 11641 consensus_queue.cc:1048] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:49.827998 10013 tablet_bootstrap.cc:654] T e9e66dacf5794b6195575556be889bce P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.856294 10013 tablet_bootstrap.cc:492] T e9e66dacf5794b6195575556be889bce P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:49.856696 10013 ts_tablet_manager.cc:1403] T e9e66dacf5794b6195575556be889bce P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.000s
I20260504 13:56:49.857100 9931 tablet_bootstrap.cc:492] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:49.857545 9931 ts_tablet_manager.cc:1403] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.002s
I20260504 13:56:49.859058 10013 raft_consensus.cc:359] T e9e66dacf5794b6195575556be889bce P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.859882 10013 raft_consensus.cc:385] T e9e66dacf5794b6195575556be889bce P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.860113 10013 raft_consensus.cc:740] T e9e66dacf5794b6195575556be889bce P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.860692 10013 consensus_queue.cc:260] T e9e66dacf5794b6195575556be889bce P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.860227 9931 raft_consensus.cc:359] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.862777 10013 ts_tablet_manager.cc:1434] T e9e66dacf5794b6195575556be889bce P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260504 13:56:49.863149 9931 raft_consensus.cc:385] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.863529 10013 tablet_bootstrap.cc:492] T b7b942037c0e45a6b3cccba288b90b00 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:49.863766 9931 raft_consensus.cc:740] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.865042 9931 consensus_queue.cc:260] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.869725 10013 tablet_bootstrap.cc:654] T b7b942037c0e45a6b3cccba288b90b00 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.878602 9931 ts_tablet_manager.cc:1434] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.021s user 0.000s sys 0.006s
I20260504 13:56:49.879881 9931 tablet_bootstrap.cc:492] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:49.887552 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eef975650610423c88ce8e4cfb0b013b" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:49.892402 9931 tablet_bootstrap.cc:654] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:49.897511 9767 leader_election.cc:304] T eef975650610423c88ce8e4cfb0b013b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:49.899986 11706 consensus_queue.cc:1048] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:49.907569 11709 consensus_queue.cc:1048] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:49.910826 9881 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:49.926915 11433 consensus_queue.cc:1048] T e29a7365caf04506b15089f10e6c1677 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:49.929657 11706 consensus_queue.cc:1048] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:49.922389 10013 tablet_bootstrap.cc:492] T b7b942037c0e45a6b3cccba288b90b00 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:49.943009 10013 ts_tablet_manager.cc:1403] T b7b942037c0e45a6b3cccba288b90b00 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.080s user 0.012s sys 0.005s
I20260504 13:56:49.954259 11624 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:49.954720 11624 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:49.959121 11624 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:49.959141 9352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 6 candidate_status { last_received { term: 4 index: 372 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:49.959803 9352 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 806407748d564439807fe5d73db9abda in term 4.
I20260504 13:56:49.960207 10814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 6 candidate_status { last_received { term: 4 index: 372 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:49.960759 10814 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 806407748d564439807fe5d73db9abda in term 5.
I20260504 13:56:49.960924 9176 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1; no voters:
I20260504 13:56:49.962638 11624 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260504 13:56:49.963129 11624 raft_consensus.cc:493] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:49.965099 11624 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 5 FOLLOWER]: Advancing to term 6
I20260504 13:56:49.972926 11624 raft_consensus.cc:515] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:49.975812 11771 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 806407748d564439807fe5d73db9abda)
I20260504 13:56:49.978394 11714 raft_consensus.cc:2749] T eef975650610423c88ce8e4cfb0b013b P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:49.981761 11771 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:49.978196 10013 raft_consensus.cc:359] T b7b942037c0e45a6b3cccba288b90b00 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.985754 10013 raft_consensus.cc:385] T b7b942037c0e45a6b3cccba288b90b00 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:49.988091 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 4 index: 372 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:49.988781 9198 raft_consensus.cc:2368] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec9f4c3a43d84fc9a420dd10723c01a1 for earlier term 5. Current term is 6.
I20260504 13:56:49.988649 10013 raft_consensus.cc:740] T b7b942037c0e45a6b3cccba288b90b00 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:49.989974 9331 leader_election.cc:400] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 806407748d564439807fe5d73db9abda (127.8.238.254:36145) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec9f4c3a43d84fc9a420dd10723c01a1 for earlier term 5. Current term is 6.
I20260504 13:56:49.990227 9331 leader_election.cc:403] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260504 13:56:49.990780 10814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 5 candidate_status { last_received { term: 4 index: 372 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:49.991499 10814 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec9f4c3a43d84fc9a420dd10723c01a1 in current term 5: Already voted for candidate 6caedf407d13459e8fd477e8b60b7dc7 in this term.
I20260504 13:56:49.992859 10013 consensus_queue.cc:260] T b7b942037c0e45a6b3cccba288b90b00 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:49.993638 11771 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:56:50.000048 11771 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 4 FOLLOWER]: Advancing to term 6
I20260504 13:56:50.004694 10013 ts_tablet_manager.cc:1434] T b7b942037c0e45a6b3cccba288b90b00 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.061s user 0.005s sys 0.001s
I20260504 13:56:50.027519 11771 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 806407748d564439807fe5d73db9abda (127.8.238.254:36145) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec9f4c3a43d84fc9a420dd10723c01a1 for earlier term 5. Current term is 6.
I20260504 13:56:50.028501 10814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 6 candidate_status { last_received { term: 4 index: 372 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7"
I20260504 13:56:50.028970 10814 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 5 FOLLOWER]: Advancing to term 6
I20260504 13:56:50.029479 11624 leader_election.cc:290] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 6 election: Requested vote from peers 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:50.029670 9352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "806407748d564439807fe5d73db9abda" candidate_term: 6 candidate_status { last_received { term: 4 index: 372 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1"
I20260504 13:56:50.035697 10814 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 806407748d564439807fe5d73db9abda in term 6.
I20260504 13:56:50.037076 9173 leader_election.cc:304] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda; no voters:
I20260504 13:56:50.039861 10013 tablet_bootstrap.cc:492] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:50.043819 9352 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 806407748d564439807fe5d73db9abda in term 6.
I20260504 13:56:50.045248 10013 tablet_bootstrap.cc:654] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:50.061662 9931 tablet_bootstrap.cc:492] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:50.062058 9931 ts_tablet_manager.cc:1403] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.182s user 0.021s sys 0.027s
I20260504 13:56:50.064580 11624 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 6 FOLLOWER]: Leader election won for term 6
I20260504 13:56:50.066596 9931 raft_consensus.cc:359] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.067328 9931 raft_consensus.cc:385] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:50.067993 9931 raft_consensus.cc:740] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:50.068818 9931 consensus_queue.cc:260] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.071061 9931 ts_tablet_manager.cc:1434] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260504 13:56:50.073064 9931 tablet_bootstrap.cc:492] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:50.073464 10013 tablet_bootstrap.cc:492] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:50.073822 10013 ts_tablet_manager.cc:1403] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.034s user 0.016s sys 0.004s
I20260504 13:56:50.077330 10013 raft_consensus.cc:359] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.077942 10013 raft_consensus.cc:385] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:50.078202 10013 raft_consensus.cc:740] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:50.078195 11711 consensus_queue.cc:1048] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:50.078831 11624 raft_consensus.cc:697] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [term 6 LEADER]: Becoming Leader. State: Replica: 806407748d564439807fe5d73db9abda, State: Running, Role: LEADER
I20260504 13:56:50.078869 10013 consensus_queue.cc:260] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.080281 10013 ts_tablet_manager.cc:1434] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260504 13:56:50.080943 9931 tablet_bootstrap.cc:654] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:50.082371 9331 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.254:36145, user_credentials={real_user=slave}} blocked reactor thread for 92727.1us
I20260504 13:56:50.087687 11624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 4.372, Last appended by leader: 372, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:50.098438 11779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 806407748d564439807fe5d73db9abda. Latest consensus state: current_term: 6 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:50.099314 11779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: This master's current role is: LEADER
I20260504 13:56:50.103030 10013 tablet_bootstrap.cc:492] T f337e9698fe64282a684e90d9c61da5a P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:50.108712 10013 tablet_bootstrap.cc:654] T f337e9698fe64282a684e90d9c61da5a P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:50.132622 10013 tablet_bootstrap.cc:492] T f337e9698fe64282a684e90d9c61da5a P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:50.133234 10013 ts_tablet_manager.cc:1403] T f337e9698fe64282a684e90d9c61da5a P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.030s user 0.017s sys 0.001s
I20260504 13:56:50.134266 9931 tablet_bootstrap.cc:492] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:50.134606 9931 ts_tablet_manager.cc:1403] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.009s
I20260504 13:56:50.135512 10013 raft_consensus.cc:359] T f337e9698fe64282a684e90d9c61da5a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.136073 10013 raft_consensus.cc:385] T f337e9698fe64282a684e90d9c61da5a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:50.136307 10013 raft_consensus.cc:740] T f337e9698fe64282a684e90d9c61da5a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:50.136798 9931 raft_consensus.cc:359] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.136899 10013 consensus_queue.cc:260] T f337e9698fe64282a684e90d9c61da5a P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.137353 9931 raft_consensus.cc:385] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:50.137593 9931 raft_consensus.cc:740] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:50.138787 11604 raft_consensus.cc:493] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:50.139226 11604 raft_consensus.cc:515] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.140748 10013 ts_tablet_manager.cc:1434] T f337e9698fe64282a684e90d9c61da5a P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260504 13:56:50.141325 11604 leader_election.cc:290] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.142802 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f337e9698fe64282a684e90d9c61da5a" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:50.143450 9672 raft_consensus.cc:2468] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:50.144080 9931 consensus_queue.cc:260] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.144701 9768 leader_election.cc:304] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:50.149252 11711 raft_consensus.cc:2804] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:50.149567 11711 raft_consensus.cc:493] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:50.149876 11711 raft_consensus.cc:3060] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.157528 11711 raft_consensus.cc:515] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.160578 9931 ts_tablet_manager.cc:1434] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.026s user 0.001s sys 0.005s
I20260504 13:56:50.161464 10013 tablet_bootstrap.cc:492] T 23ff0d52a90c4056880190dbd936cb8e P 7b79fe7dd8934dd996393a2d96eabf00: Bootstrap starting.
I20260504 13:56:50.161971 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f337e9698fe64282a684e90d9c61da5a" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:50.162408 9672 raft_consensus.cc:3060] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.163235 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f337e9698fe64282a684e90d9c61da5a" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:50.163810 9531 raft_consensus.cc:2468] T f337e9698fe64282a684e90d9c61da5a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:50.167279 11711 leader_election.cc:290] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.168336 10013 tablet_bootstrap.cc:654] T 23ff0d52a90c4056880190dbd936cb8e P 7b79fe7dd8934dd996393a2d96eabf00: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:50.168745 11706 raft_consensus.cc:493] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:50.168975 9672 raft_consensus.cc:2468] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:50.169145 11706 raft_consensus.cc:515] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.169994 9768 leader_election.cc:304] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:50.170785 11711 raft_consensus.cc:2804] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:50.170819 11706 leader_election.cc:290] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.171325 11711 raft_consensus.cc:697] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:50.172842 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e70772b7ce4f27a941918aaf66a865" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:50.173362 9672 raft_consensus.cc:2468] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:50.174281 9768 leader_election.cc:304] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:50.174929 11604 raft_consensus.cc:2804] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:50.172434 11711 consensus_queue.cc:237] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.175411 11604 raft_consensus.cc:493] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:50.175740 11604 raft_consensus.cc:3060] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.176060 9931 tablet_bootstrap.cc:492] T 564d0fafb49a4b4bb3bdb240c74c3410 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:50.189043 11604 raft_consensus.cc:515] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.191813 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e70772b7ce4f27a941918aaf66a865" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:50.192313 9672 raft_consensus.cc:3060] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.199239 9672 raft_consensus.cc:2468] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:50.200111 9768 leader_election.cc:304] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:50.200798 11714 raft_consensus.cc:2804] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:50.201612 11714 raft_consensus.cc:697] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:50.201866 11732 raft_consensus.cc:493] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:50.202265 11732 raft_consensus.cc:515] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.202479 11714 consensus_queue.cc:237] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.204674 10013 tablet_bootstrap.cc:492] T 23ff0d52a90c4056880190dbd936cb8e P 7b79fe7dd8934dd996393a2d96eabf00: No bootstrap required, opened a new log
I20260504 13:56:50.204887 11732 leader_election.cc:290] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.205070 10013 ts_tablet_manager.cc:1403] T 23ff0d52a90c4056880190dbd936cb8e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.000s
I20260504 13:56:50.205106 9675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd635a863f7a41d087947f619e41f95c" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:50.132951 9181 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4]). Expected version 1 got 0
I20260504 13:56:50.205631 9675 raft_consensus.cc:2468] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:50.206591 9473 leader_election.cc:304] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:50.207300 11732 raft_consensus.cc:2804] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:50.207713 11732 raft_consensus.cc:493] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:50.209372 11732 raft_consensus.cc:3060] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.214728 9588 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:50.216903 10013 raft_consensus.cc:359] T 23ff0d52a90c4056880190dbd936cb8e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.217475 10013 raft_consensus.cc:385] T 23ff0d52a90c4056880190dbd936cb8e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:50.217718 10013 raft_consensus.cc:740] T 23ff0d52a90c4056880190dbd936cb8e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Initialized, Role: FOLLOWER
I20260504 13:56:50.225878 9931 tablet_bootstrap.cc:654] T 564d0fafb49a4b4bb3bdb240c74c3410 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:50.218241 10013 consensus_queue.cc:260] T 23ff0d52a90c4056880190dbd936cb8e P 7b79fe7dd8934dd996393a2d96eabf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.228386 10013 ts_tablet_manager.cc:1434] T 23ff0d52a90c4056880190dbd936cb8e P 7b79fe7dd8934dd996393a2d96eabf00: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20260504 13:56:50.228520 11433 raft_consensus.cc:493] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:50.228896 11433 raft_consensus.cc:515] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.230576 11433 leader_election.cc:290] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.231868 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564d0fafb49a4b4bb3bdb240c74c3410" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:50.232445 9672 raft_consensus.cc:2468] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:50.233517 9473 leader_election.cc:304] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:50.234220 11433 raft_consensus.cc:2804] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:50.234575 11433 raft_consensus.cc:493] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:50.234912 11433 raft_consensus.cc:3060] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.237130 11604 leader_election.cc:290] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.243367 11433 raft_consensus.cc:515] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.245656 9931 tablet_bootstrap.cc:492] T 564d0fafb49a4b4bb3bdb240c74c3410 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:50.246076 9931 ts_tablet_manager.cc:1403] T 564d0fafb49a4b4bb3bdb240c74c3410 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.071s user 0.011s sys 0.000s
I20260504 13:56:50.249392 11480 raft_consensus.cc:493] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:50.249782 11480 raft_consensus.cc:515] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.250499 9931 raft_consensus.cc:359] T 564d0fafb49a4b4bb3bdb240c74c3410 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.251104 9931 raft_consensus.cc:385] T 564d0fafb49a4b4bb3bdb240c74c3410 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:50.251262 11480 leader_election.cc:290] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.251380 9931 raft_consensus.cc:740] T 564d0fafb49a4b4bb3bdb240c74c3410 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:50.252342 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e70772b7ce4f27a941918aaf66a865" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:50.253275 9672 raft_consensus.cc:2393] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 1: Already voted for candidate d4051d2bf0bf410eb8acd107560df888 in this term.
I20260504 13:56:50.253820 9931 consensus_queue.cc:260] T 564d0fafb49a4b4bb3bdb240c74c3410 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.257054 9931 ts_tablet_manager.cc:1434] T 564d0fafb49a4b4bb3bdb240c74c3410 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260504 13:56:50.269754 11706 raft_consensus.cc:493] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:50.270144 11706 raft_consensus.cc:515] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.271826 11706 leader_election.cc:290] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.273424 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9e66dacf5794b6195575556be889bce" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:50.273977 9672 raft_consensus.cc:2468] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:50.275063 9768 leader_election.cc:304] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:50.275733 11604 raft_consensus.cc:493] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:50.276108 11604 raft_consensus.cc:515] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.277791 11604 leader_election.cc:290] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.278252 11604 raft_consensus.cc:2804] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:50.278559 11604 raft_consensus.cc:493] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:50.278891 11604 raft_consensus.cc:3060] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.282760 9675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b942037c0e45a6b3cccba288b90b00" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:50.283988 9675 raft_consensus.cc:2468] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:50.285166 9181 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
I20260504 13:56:50.285223 9768 leader_election.cc:304] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:50.285952 11706 raft_consensus.cc:2804] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:50.286269 11706 raft_consensus.cc:493] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:50.286628 11706 raft_consensus.cc:3060] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.289026 11604 raft_consensus.cc:515] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.291927 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9e66dacf5794b6195575556be889bce" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:50.293792 11706 raft_consensus.cc:515] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.295049 9672 raft_consensus.cc:3060] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.301710 9672 raft_consensus.cc:2468] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:50.302232 11711 raft_consensus.cc:493] T 6677a93d01d54f15a506ae1474bcb501 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:50.302657 11711 raft_consensus.cc:515] T 6677a93d01d54f15a506ae1474bcb501 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.304401 11711 leader_election.cc:290] T 6677a93d01d54f15a506ae1474bcb501 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.303745 9675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b942037c0e45a6b3cccba288b90b00" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:50.305219 11604 leader_election.cc:290] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.305475 9675 raft_consensus.cc:3060] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.313791 9768 leader_election.cc:304] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:50.315523 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6677a93d01d54f15a506ae1474bcb501" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:50.316036 11604 raft_consensus.cc:2804] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:50.316272 9672 raft_consensus.cc:2393] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4051d2bf0bf410eb8acd107560df888 in current term 1: Already voted for candidate 7b79fe7dd8934dd996393a2d96eabf00 in this term.
I20260504 13:56:50.316471 11604 raft_consensus.cc:697] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:50.317200 11604 consensus_queue.cc:237] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.329437 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f337e9698fe64282a684e90d9c61da5a" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:50.331511 9539 raft_consensus.cc:3060] T f337e9698fe64282a684e90d9c61da5a P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.342250 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e70772b7ce4f27a941918aaf66a865" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:50.342806 9531 raft_consensus.cc:2468] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:50.343720 9675 raft_consensus.cc:2468] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:50.344566 11706 leader_election.cc:290] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.344807 9768 leader_election.cc:304] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:50.346535 11706 raft_consensus.cc:2804] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:50.346921 11706 raft_consensus.cc:697] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:50.349148 11711 raft_consensus.cc:493] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:50.351379 11711 raft_consensus.cc:515] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.353192 11711 leader_election.cc:290] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.354023 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd635a863f7a41d087947f619e41f95c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:50.354588 9672 raft_consensus.cc:2468] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:50.355597 9768 leader_election.cc:304] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:50.357028 11711 raft_consensus.cc:2804] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:50.357347 11711 raft_consensus.cc:493] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:50.357672 11711 raft_consensus.cc:3060] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.361315 11709 raft_consensus.cc:493] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:50.361791 11709 raft_consensus.cc:515] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.354583 11706 consensus_queue.cc:237] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.363768 11709 leader_election.cc:290] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.367420 11433 leader_election.cc:290] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.369578 9531 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e70772b7ce4f27a941918aaf66a865" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:50.370666 9531 raft_consensus.cc:3060] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.379034 11711 raft_consensus.cc:515] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.393184 9539 raft_consensus.cc:2468] T f337e9698fe64282a684e90d9c61da5a P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:50.397953 11711 leader_election.cc:290] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.406603 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564d0fafb49a4b4bb3bdb240c74c3410" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:50.407079 9672 raft_consensus.cc:3060] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.415434 9672 raft_consensus.cc:2468] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:50.416908 9473 leader_election.cc:304] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:50.417542 11433 raft_consensus.cc:2804] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:50.417953 11433 raft_consensus.cc:697] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:50.418668 11433 consensus_queue.cc:237] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.421958 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9e66dacf5794b6195575556be889bce" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:50.423348 9539 raft_consensus.cc:2468] T e9e66dacf5794b6195575556be889bce P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:50.422559 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b942037c0e45a6b3cccba288b90b00" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:50.424072 9533 raft_consensus.cc:2468] T b7b942037c0e45a6b3cccba288b90b00 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:50.431756 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9e66dacf5794b6195575556be889bce" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:50.432219 9539 raft_consensus.cc:3060] T e9e66dacf5794b6195575556be889bce P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.433828 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b942037c0e45a6b3cccba288b90b00" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:50.434255 9530 raft_consensus.cc:3060] T b7b942037c0e45a6b3cccba288b90b00 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.436708 9533 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6677a93d01d54f15a506ae1474bcb501" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:50.437254 11732 raft_consensus.cc:515] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.441474 9530 raft_consensus.cc:2468] T b7b942037c0e45a6b3cccba288b90b00 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:50.442147 9675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd635a863f7a41d087947f619e41f95c" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:50.442623 9675 raft_consensus.cc:3060] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.447153 9683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd635a863f7a41d087947f619e41f95c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:50.451103 9672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ff0d52a90c4056880190dbd936cb8e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:50.451627 9672 raft_consensus.cc:2468] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:50.460837 11790 consensus_queue.cc:1048] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:50.461311 9768 leader_election.cc:304] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:50.462343 11790 consensus_queue.cc:1048] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:50.463440 11790 raft_consensus.cc:2804] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:56:50.463738 11790 raft_consensus.cc:493] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:56:50.464030 11790 raft_consensus.cc:3060] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.469506 9675 raft_consensus.cc:2468] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:50.454452 9539 raft_consensus.cc:2468] T e9e66dacf5794b6195575556be889bce P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:50.479401 9473 leader_election.cc:304] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00; no voters:
I20260504 13:56:50.480336 11480 raft_consensus.cc:2804] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:50.482281 9535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd635a863f7a41d087947f619e41f95c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:50.483644 9539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ff0d52a90c4056880190dbd936cb8e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00" is_pre_election: true
I20260504 13:56:50.484292 9539 raft_consensus.cc:2468] T 23ff0d52a90c4056880190dbd936cb8e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 0.
I20260504 13:56:50.488785 11790 raft_consensus.cc:515] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.492714 9684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ff0d52a90c4056880190dbd936cb8e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6"
I20260504 13:56:50.493196 9684 raft_consensus.cc:3060] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.402855 9531 raft_consensus.cc:2468] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:50.499619 9684 raft_consensus.cc:2468] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:50.499894 9181 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 2ddf890c17964e7f90409ebe3c4034ef (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]). Expected version 2 got 0
I20260504 13:56:50.500777 9768 leader_election.cc:304] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888; no voters:
I20260504 13:56:50.502100 11604 raft_consensus.cc:2804] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:56:50.502450 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd635a863f7a41d087947f619e41f95c" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:50.502537 11604 raft_consensus.cc:697] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Becoming Leader. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Running, Role: LEADER
I20260504 13:56:50.503314 11604 consensus_queue.cc:237] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.508090 11732 leader_election.cc:290] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 election: Requested vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.516592 9530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ff0d52a90c4056880190dbd936cb8e" candidate_uuid: "d4051d2bf0bf410eb8acd107560df888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b79fe7dd8934dd996393a2d96eabf00"
I20260504 13:56:50.517092 9530 raft_consensus.cc:3060] T 23ff0d52a90c4056880190dbd936cb8e P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.519213 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd635a863f7a41d087947f619e41f95c" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:50.519845 9830 raft_consensus.cc:2393] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 1: Already voted for candidate d4051d2bf0bf410eb8acd107560df888 in this term.
I20260504 13:56:50.521624 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564d0fafb49a4b4bb3bdb240c74c3410" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:50.522071 9830 raft_consensus.cc:2468] T 564d0fafb49a4b4bb3bdb240c74c3410 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 0.
I20260504 13:56:50.522832 9530 raft_consensus.cc:2468] T 23ff0d52a90c4056880190dbd936cb8e P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4051d2bf0bf410eb8acd107560df888 in term 1.
I20260504 13:56:50.525045 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564d0fafb49a4b4bb3bdb240c74c3410" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:50.525807 9830 raft_consensus.cc:3060] T 564d0fafb49a4b4bb3bdb240c74c3410 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.533525 11480 raft_consensus.cc:697] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Becoming Leader. State: Replica: 7b79fe7dd8934dd996393a2d96eabf00, State: Running, Role: LEADER
I20260504 13:56:50.537739 11604 consensus_queue.cc:1048] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:50.545918 9830 raft_consensus.cc:2468] T 564d0fafb49a4b4bb3bdb240c74c3410 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b79fe7dd8934dd996393a2d96eabf00 in term 1.
I20260504 13:56:50.534991 11480 consensus_queue.cc:237] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.547698 9833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e70772b7ce4f27a941918aaf66a865" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:50.552455 9181 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]). Expected version 1 got 0
I20260504 13:56:50.553709 9181 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]). Expected version 1 got 0
I20260504 13:56:50.555291 9181 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 3 got 0
I20260504 13:56:50.558890 11790 leader_election.cc:290] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Requested vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.584867 11706 consensus_queue.cc:1048] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:50.586645 9767 leader_election.cc:304] T 6677a93d01d54f15a506ae1474bcb501 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:50.587615 11706 raft_consensus.cc:3060] T 6677a93d01d54f15a506ae1474bcb501 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.596657 11795 consensus_queue.cc:1048] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:50.624253 11706 raft_consensus.cc:2749] T 6677a93d01d54f15a506ae1474bcb501 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:50.632169 9830 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd635a863f7a41d087947f619e41f95c" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888"
I20260504 13:56:50.689818 9830 raft_consensus.cc:2393] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7b79fe7dd8934dd996393a2d96eabf00 in current term 1: Already voted for candidate d4051d2bf0bf410eb8acd107560df888 in this term.
I20260504 13:56:50.648011 9767 leader_election.cc:304] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:50.691524 11604 raft_consensus.cc:2749] T bd635a863f7a41d087947f619e41f95c P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260504 13:56:50.686226 11790 raft_consensus.cc:493] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:50.695513 11790 raft_consensus.cc:515] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.697265 11790 leader_election.cc:290] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
W20260504 13:56:50.698740 9768 leader_election.cc:343] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Not found: Tablet not found: 41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:50.705519 10814 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 6 FOLLOWER]: Refusing update from remote peer 806407748d564439807fe5d73db9abda: Log matching property violated. Preceding OpId in replica: term: 4 index: 371. Preceding OpId from leader: term: 6 index: 373. (index mismatch)
I20260504 13:56:50.724792 9531 raft_consensus.cc:3060] T 7989338bc4bf4bad8fb8fe37587d0a16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:56:50.756376 9736 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:50.762193 11803 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [LEADER]: Connected to new peer: Peer: permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 373, Last known committed idx: 371, Time since last communication: 0.001s
I20260504 13:56:50.795816 9352 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 6 FOLLOWER]: Refusing update from remote peer 806407748d564439807fe5d73db9abda: Log matching property violated. Preceding OpId in replica: term: 4 index: 372. Preceding OpId from leader: term: 6 index: 373. (index mismatch)
I20260504 13:56:50.797245 11802 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 373, Last known committed idx: 372, Time since last communication: 0.001s
I20260504 13:56:50.836105 11806 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 806407748d564439807fe5d73db9abda. Latest consensus state: current_term: 6 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:50.836797 11806 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:56:50.857244 11806 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:50.857933 11806 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:56:50.862442 11805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 806407748d564439807fe5d73db9abda. Latest consensus state: current_term: 6 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:50.863127 11805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:56:50.875113 11805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:50.875746 11805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:56:50.854341 9180 catalog_manager.cc:5671] T a2bfb681e6824e9f9bdcbea23061d57a P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:50.897863 9180 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
I20260504 13:56:50.908145 11790 consensus_queue.cc:1048] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:50.908730 9180 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f]). Expected version 3 got 0
I20260504 13:56:50.909415 11790 consensus_queue.cc:1048] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:50.910861 9180 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4]). Expected version 1 got 0
W20260504 13:56:50.915143 9767 leader_election.cc:343] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Not found: Tablet not found: 41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:50.915621 9767 leader_election.cc:304] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4051d2bf0bf410eb8acd107560df888; no voters: 5beceb6317444247b31b44a6581707c6, 7b79fe7dd8934dd996393a2d96eabf00
I20260504 13:56:50.918875 11714 raft_consensus.cc:2749] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:50.945101 9180 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 2ddf890c17964e7f90409ebe3c4034ef (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]). Expected version 2 got 0
I20260504 13:56:50.945890 9180 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]). Expected version 1 got 0
I20260504 13:56:50.947211 9180 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]). Expected version 1 got 0
I20260504 13:56:50.963743 9187 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e] alter complete (version 1)
I20260504 13:56:50.966284 11661 raft_consensus.cc:493] T 7989338bc4bf4bad8fb8fe37587d0a16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:50.966739 11661 raft_consensus.cc:515] T 7989338bc4bf4bad8fb8fe37587d0a16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:50.966914 11803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:50.967588 9189 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5] alter complete (version 4)
I20260504 13:56:50.967839 11803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: This master's current role is: LEADER
I20260504 13:56:50.968578 11802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "806407748d564439807fe5d73db9abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } } }
I20260504 13:56:50.969173 11802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda [sys.catalog]: This master's current role is: LEADER
I20260504 13:56:50.969993 11661 leader_election.cc:290] T 7989338bc4bf4bad8fb8fe37587d0a16 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:50.975687 9682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7989338bc4bf4bad8fb8fe37587d0a16" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5beceb6317444247b31b44a6581707c6" is_pre_election: true
I20260504 13:56:50.982939 9474 leader_election.cc:304] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:50.998374 9183 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c] alter complete (version 1)
I20260504 13:56:50.998867 11795 raft_consensus.cc:2749] T e0e70772b7ce4f27a941918aaf66a865 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:56:51.037978 9184 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568] alter complete (version 1)
I20260504 13:56:51.047314 9797 tablet_service.cc:1558] Processing DeleteTablet for tablet da61c91e10264fa49fe397d9a9b014bb with delete_type TABLET_DATA_DELETED (replaced by tablet 6323e1c10e7643f8a28f04e285f620e9) from {username='slave'} at 127.0.0.1:40598
I20260504 13:56:51.062084 9801 tablet_service.cc:1558] Processing DeleteTablet for tablet 8bb77106b48a45a2aaa938dc474fa8bd with delete_type TABLET_DATA_DELETED (replaced by tablet f337e9698fe64282a684e90d9c61da5a) from {username='slave'} at 127.0.0.1:40598
I20260504 13:56:51.062170 11732 consensus_queue.cc:1048] T 6677a93d01d54f15a506ae1474bcb501 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:51.066195 9183 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38] alter complete (version 1)
I20260504 13:56:51.087647 9805 tablet_service.cc:1558] Processing DeleteTablet for tablet 8e4e583afc804212bece2f727411cc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:23 UTC) from {username='slave'} at 127.0.0.1:40598
I20260504 13:56:51.089083 9805 tablet_service.cc:1558] Processing DeleteTablet for tablet 41d41f8ba6dc4bb18116cb2b2dfabff0 with delete_type TABLET_DATA_DELETED (replaced by tablet eef975650610423c88ce8e4cfb0b013b) from {username='slave'} at 127.0.0.1:40598
I20260504 13:56:51.089807 9801 tablet_service.cc:1558] Processing DeleteTablet for tablet 21d6358b451d4b8d87a22e30e0780f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:25 UTC) from {username='slave'} at 127.0.0.1:40598
I20260504 13:56:51.094650 11818 tablet_replica.cc:333] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:51.095692 11818 raft_consensus.cc:2243] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:51.096424 11818 raft_consensus.cc:2272] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260504 13:56:51.100346 9184 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 92480654807d495b879576b0195871bd consensus state has no leader
W20260504 13:56:51.102458 9184 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:33659 (request call id 242) took 9507 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.102794 9184 rpcz_store.cc:269] 0504 13:56:41.595006 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:41.595191 (+ 185us) service_pool.cc:224] Handling call
0504 13:56:51.102392 (+9507201us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:50.999003 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:50.999524 (+ 521us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:50.999551 (+ 27us) write_op.cc:435] Acquired schema lock
0504 13:56:50.999565 (+ 14us) tablet.cc:662] Decoding operations
0504 13:56:50.999852 (+ 287us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:50.999900 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:50.999924 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:51.000076 (+ 152us) tablet.cc:701] Row locks acquired
0504 13:56:51.000088 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:51.000220 (+ 132us) write_op.cc:270] Start()
0504 13:56:51.000307 (+ 87us) write_op.cc:276] Timestamp: P: 1777903011000141 usec, L: 0
0504 13:56:51.000322 (+ 15us) op_driver.cc:348] REPLICATION: starting
0504 13:56:51.000825 (+ 503us) log.cc:844] Serialized 356 byte log entry
0504 13:56:51.006357 (+ 5532us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b5400020920. stack: 00007efe36c52c83 00007efe36c52b80 00007efe367c3574 000000000050e094 000000000050e029 00007efe3dcaa951 00007efe3dcc1f06 00007efe3dcbb87a 00007efe3dcbaaf4 00007efe3dc9b28a 00007efe3dca2133 00007efe3dca20b9 00007efe3dca2049 00007efe3dca2011 00007efe3dca11dd 00000000005b3bc4
0504 13:56:51.014940 (+ 8583us) op_driver.cc:464] REPLICATION: finished
0504 13:56:51.035852 (+ 20912us) write_op.cc:301] APPLY: starting
0504 13:56:51.035910 (+ 58us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:51.035997 (+ 87us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:51.036018 (+ 21us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:51.036258 (+ 240us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:51.036407 (+ 149us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:51.036452 (+ 45us) write_op.cc:312] APPLY: finished
0504 13:56:51.036660 (+ 208us) log.cc:844] Serialized 23 byte log entry
0504 13:56:51.037359 (+ 699us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:51.037420 (+ 61us) write_op.cc:454] Released schema lock
0504 13:56:51.037452 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":65048,"reported_tablets":99,"child_traces":[["op",{"apply.queue_time_us":20753,"apply.run_cpu_time_us":1315,"apply.run_wall_time_us":1853,"num_ops":1,"prepare.queue_time_us":758,"prepare.run_cpu_time_us":2187,"prepare.run_wall_time_us":2187,"raft.queue_time_us":1954,"raft.run_cpu_time_us":3995,"raft.run_wall_time_us":5969,"replication_time_us":14523,"spinlock_wait_cycles":5951872,"thread_start_us":39455,"threads_started":1}]]}
I20260504 13:56:51.134480 9505 tablet_service.cc:1558] Processing DeleteTablet for tablet 8e4e583afc804212bece2f727411cc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:23 UTC) from {username='slave'} at 127.0.0.1:40114
I20260504 13:56:51.136127 9517 tablet_service.cc:1558] Processing DeleteTablet for tablet 21d6358b451d4b8d87a22e30e0780f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:25 UTC) from {username='slave'} at 127.0.0.1:40114
W20260504 13:56:51.139678 9184 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:39697 (request call id 230) took 6302 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.139972 9184 rpcz_store.cc:269] 0504 13:56:44.837547 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:51.139376 (+6301829us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:51.139617 (+ 241us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 13:56:51.140676 9184 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:45997 (request call id 273) took 5280 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.140964 9184 rpcz_store.cc:269] 0504 13:56:45.860533 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:51.140537 (+5280004us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:51.140629 (+ 92us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 13:56:51.141692 9184 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:33659 (request call id 245) took 5186 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.142000 9184 rpcz_store.cc:269] 0504 13:56:45.955361 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:51.141554 (+5186193us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:51.141649 (+ 95us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 13:56:51.143380 9184 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:39697 (request call id 231) took 5111 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.143704 9184 rpcz_store.cc:269] 0504 13:56:46.031755 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:51.143110 (+5111355us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:51.143332 (+ 222us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 13:56:51.144353 9184 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:45997 (request call id 274) took 3873 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.144809 9184 rpcz_store.cc:269] 0504 13:56:47.270665 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:51.144230 (+3873565us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:51.144311 (+ 81us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 13:56:51.145596 9184 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:33659 (request call id 246) took 3866 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.145865 9184 rpcz_store.cc:269] 0504 13:56:47.279579 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:51.145475 (+3865896us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:51.145555 (+ 80us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 13:56:51.150866 9174 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92480654807d495b879576b0195871bd consensus state has no leader
W20260504 13:56:51.159483 9184 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:39697 (request call id 232) took 3588 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.159832 9184 rpcz_store.cc:269] 0504 13:56:47.570755 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:51.159333 (+3588578us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:51.159430 (+ 97us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 13:56:51.171437 9184 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:45997 (request call id 275) took 2563 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.171753 9184 rpcz_store.cc:269] 0504 13:56:48.608186 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:51.171273 (+2563087us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:51.171382 (+ 109us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 13:56:51.172451 9184 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:33659 (request call id 247) took 2239 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.172734 9184 rpcz_store.cc:269] 0504 13:56:48.933286 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:51.172301 (+2239015us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:51.172403 (+ 102us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 13:56:51.178406 9184 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:39697 (request call id 233) took 1968 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.178694 9184 rpcz_store.cc:269] 0504 13:56:49.210286 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:51.178228 (+1967942us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:51.178314 (+ 86us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260504 13:56:51.179354 9184 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:45997 (request call id 276) took 1419 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.179646 9184 rpcz_store.cc:269] 0504 13:56:49.760270 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:51.179217 (+1418947us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:51.179302 (+ 85us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260504 13:56:51.184689 9517 tablet_service.cc:1558] Processing DeleteTablet for tablet da61c91e10264fa49fe397d9a9b014bb with delete_type TABLET_DATA_DELETED (replaced by tablet 6323e1c10e7643f8a28f04e285f620e9) from {username='slave'} at 127.0.0.1:40114
I20260504 13:56:51.141042 11714 consensus_queue.cc:1048] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.052s
I20260504 13:56:51.205312 9186 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469] alter complete (version 1)
W20260504 13:56:51.206712 9187 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:33659 (request call id 241) took 11695 ms (client timeout 1000 ms). Trace:
I20260504 13:56:51.127254 11818 ts_tablet_manager.cc:1916] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:51.228060 11831 tablet.cc:1724] T a862a43c5ac443d9bac98e0472ca23b2 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260504 13:56:51.241873 11832 tablet.cc:1724] T a862a43c5ac443d9bac98e0472ca23b2 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20260504 13:56:51.209329 9189 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:39697 (request call id 227) took 11421 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.256815 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 175005us
I20260504 13:56:51.257040 11826 tablet_replica.cc:333] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:51.276433 11706 consensus_queue.cc:1048] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:51.291268 9176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.252:44781, user_credentials={real_user=slave}} blocked reactor thread for 213798us
I20260504 13:56:51.293357 11831 tablet.cc:1724] T 5cb30835150949f39f6e7e7563381763 P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:51.306318 9881 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
W20260504 13:56:51.306842 9184 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:33659 (request call id 248) took 1005 ms (client timeout 1000 ms). Trace:
I20260504 13:56:51.316913 11795 consensus_queue.cc:1048] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:51.317183 11826 raft_consensus.cc:2243] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:51.318030 11826 raft_consensus.cc:2272] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:51.320591 11826 ts_tablet_manager.cc:1916] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260504 13:56:51.324420 9184 rpcz_store.cc:269] 0504 13:56:50.301893 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:51.180180 (+878287us) service_pool.cc:224] Handling call
0504 13:56:51.306771 (+126591us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:51.353678 11826 ts_tablet_manager.cc:1929] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:51.354184 11826 log.cc:1199] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/8e4e583afc804212bece2f727411cc14
I20260504 13:56:51.359725 11826 ts_tablet_manager.cc:1950] T 8e4e583afc804212bece2f727411cc14 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:51.362829 11826 tablet_replica.cc:333] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:51.363955 11826 raft_consensus.cc:2243] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:51.364571 11826 raft_consensus.cc:2272] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:51.365661 11844 tablet.cc:1724] T 5cb30835150949f39f6e7e7563381763 P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:51.369143 11826 ts_tablet_manager.cc:1916] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:51.368882 11845 tablet.cc:1724] T c47b6e3b4d0c45c28384f7aae6b749de P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260504 13:56:51.245178 9189 rpcz_store.cc:269] 0504 13:56:39.794087 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:39.794291 (+ 204us) service_pool.cc:224] Handling call
0504 13:56:39.849440 (+ 55149us) spinlock_profiling.cc:243] Waited 42.9 ms on lock 0x7b54000203d4. stack: 00007efe36c52c83 00007efe36c52b80 00007efe367c3574 000000000050e094 000000000050e029 00007efe3dcab470 00007efe3dd4a31a 00007efe3fa5e46c 00007efe3fb583ee 00007efe3d75f4a4 00007efe3d75f432 00007efe3d75f361 00007efe3d75f2dc 00007efe3d75e592 00007efe3ba34bcc 00007efe3ba34006
0504 13:56:41.569100 (+1719660us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:41.569127 (+ 27us) write_op.cc:454] Released schema lock
0504 13:56:51.209262 (+9640135us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:41.552342 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:41.552835 (+ 493us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:41.552863 (+ 28us) write_op.cc:435] Acquired schema lock
0504 13:56:41.552877 (+ 14us) tablet.cc:662] Decoding operations
0504 13:56:41.553245 (+ 368us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:41.553307 (+ 62us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:41.553344 (+ 37us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:41.553552 (+ 208us) tablet.cc:701] Row locks acquired
0504 13:56:41.553570 (+ 18us) write_op.cc:260] PREPARE: finished
0504 13:56:41.553731 (+ 161us) write_op.cc:270] Start()
0504 13:56:41.553821 (+ 90us) write_op.cc:276] Timestamp: P: 1777903001553671 usec, L: 0
0504 13:56:41.553834 (+ 13us) op_driver.cc:348] REPLICATION: starting
0504 13:56:41.554327 (+ 493us) log.cc:844] Serialized 533 byte log entry
0504 13:56:41.565556 (+ 11229us) op_driver.cc:464] REPLICATION: finished
0504 13:56:41.565950 (+ 394us) write_op.cc:301] APPLY: starting
0504 13:56:41.566004 (+ 54us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:41.566092 (+ 88us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:41.566172 (+ 80us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:41.566323 (+ 151us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:41.566500 (+ 177us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:41.566564 (+ 64us) write_op.cc:312] APPLY: finished
0504 13:56:41.566718 (+ 154us) log.cc:844] Serialized 23 byte log entry
0504 13:56:41.566848 (+ 130us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:41.566907 (+ 59us) write_op.cc:454] Released schema lock
0504 13:56:41.566937 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0504 13:56:41.839449 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:41.839848 (+ 399us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:41.839873 (+ 25us) write_op.cc:435] Acquired schema lock
0504 13:56:41.839887 (+ 14us) tablet.cc:662] Decoding operations
0504 13:56:41.840165 (+ 278us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:41.840204 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:41.840228 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:41.840380 (+ 152us) tablet.cc:701] Row locks acquired
0504 13:56:41.840395 (+ 15us) write_op.cc:260] PREPARE: finished
0504 13:56:41.840516 (+ 121us) write_op.cc:270] Start()
0504 13:56:41.840639 (+ 123us) write_op.cc:276] Timestamp: P: 1777903001840440 usec, L: 0
0504 13:56:41.840665 (+ 26us) op_driver.cc:348] REPLICATION: starting
0504 13:56:41.841305 (+ 640us) log.cc:844] Serialized 356 byte log entry
0504 13:56:41.858716 (+ 17411us) op_driver.cc:464] REPLICATION: finished
0504 13:56:41.860223 (+ 1507us) write_op.cc:301] APPLY: starting
0504 13:56:41.860276 (+ 53us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:41.860375 (+ 99us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:41.860457 (+ 82us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:41.860598 (+ 141us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:41.860733 (+ 135us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:41.860774 (+ 41us) write_op.cc:312] APPLY: finished
0504 13:56:41.860924 (+ 150us) log.cc:844] Serialized 23 byte log entry
0504 13:56:41.861048 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:41.861102 (+ 54us) write_op.cc:454] Released schema lock
0504 13:56:41.861129 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:41.861372 (+ 243us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:41.861396 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":178060,"reported_tablets":97,"spinlock_wait_cycles":98687744,"child_traces":[["op",{"apply.queue_time_us":249,"apply.run_cpu_time_us":1151,"apply.run_wall_time_us":1151,"num_ops":1,"prepare.queue_time_us":2324,"prepare.run_cpu_time_us":2275,"prepare.run_wall_time_us":16311,"raft.queue_time_us":116,"raft.run_cpu_time_us":604,"raft.run_wall_time_us":603,"replication_time_us":9307}],["op",{"apply.queue_time_us":1368,"apply.run_cpu_time_us":1108,"apply.run_wall_time_us":1109,"num_ops":1,"prepare.queue_time_us":7747,"prepare.run_cpu_time_us":2224,"prepare.run_wall_time_us":2223,"raft.queue_time_us":110,"raft.run_cpu_time_us":595,"raft.run_wall_time_us":595,"replication_time_us":17949,"spinlock_wait_cycles":30592}]]}
I20260504 13:56:51.383502 11711 consensus_queue.cc:1048] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260504 13:56:51.233033 9187 rpcz_store.cc:269] 0504 13:56:39.510858 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:39.510983 (+ 125us) service_pool.cc:224] Handling call
0504 13:56:41.545509 (+2034526us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:41.545547 (+ 38us) write_op.cc:454] Released schema lock
0504 13:56:41.684255 (+138708us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:41.684282 (+ 27us) write_op.cc:454] Released schema lock
0504 13:56:51.206637 (+9522355us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:41.098884 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:41.099884 (+ 1000us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:41.099911 (+ 27us) write_op.cc:435] Acquired schema lock
0504 13:56:41.099926 (+ 15us) tablet.cc:662] Decoding operations
0504 13:56:41.100836 (+ 910us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:41.100888 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:41.100917 (+ 29us) tablet.cc:685] Acquiring locks for 11 operations
0504 13:56:41.120227 (+ 19310us) tablet.cc:701] Row locks acquired
0504 13:56:41.120247 (+ 20us) write_op.cc:260] PREPARE: finished
0504 13:56:41.120375 (+ 128us) write_op.cc:270] Start()
0504 13:56:41.120467 (+ 92us) write_op.cc:276] Timestamp: P: 1777903001120316 usec, L: 0
0504 13:56:41.120489 (+ 22us) op_driver.cc:348] REPLICATION: starting
0504 13:56:41.121080 (+ 591us) log.cc:844] Serialized 4334 byte log entry
0504 13:56:41.191670 (+ 70590us) op_driver.cc:464] REPLICATION: finished
0504 13:56:41.228592 (+ 36922us) write_op.cc:301] APPLY: starting
0504 13:56:41.228655 (+ 63us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:41.228896 (+ 241us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:41.229022 (+ 126us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:41.283998 (+ 54976us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:41.284241 (+ 243us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0504 13:56:41.284300 (+ 59us) write_op.cc:312] APPLY: finished
0504 13:56:41.284684 (+ 384us) log.cc:844] Serialized 83 byte log entry
0504 13:56:41.284831 (+ 147us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:41.284953 (+ 122us) write_op.cc:454] Released schema lock
0504 13:56:41.284990 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:41.394739 (+109749us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:41.394765 (+ 26us) write_op.cc:454] Released schema lock
Related trace 'op':
0504 13:56:41.341082 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:41.341554 (+ 472us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:41.341580 (+ 26us) write_op.cc:435] Acquired schema lock
0504 13:56:41.341592 (+ 12us) tablet.cc:662] Decoding operations
0504 13:56:41.341861 (+ 269us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:41.341905 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:41.341930 (+ 25us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:41.342093 (+ 163us) tablet.cc:701] Row locks acquired
0504 13:56:41.342105 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:41.342220 (+ 115us) write_op.cc:270] Start()
0504 13:56:41.342304 (+ 84us) write_op.cc:276] Timestamp: P: 1777903001342146 usec, L: 0
0504 13:56:41.342322 (+ 18us) op_driver.cc:348] REPLICATION: starting
0504 13:56:41.342816 (+ 494us) log.cc:844] Serialized 356 byte log entry
0504 13:56:41.363701 (+ 20885us) op_driver.cc:464] REPLICATION: finished
0504 13:56:41.378069 (+ 14368us) write_op.cc:301] APPLY: starting
0504 13:56:41.378131 (+ 62us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:41.378222 (+ 91us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:41.378342 (+ 120us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:41.378510 (+ 168us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:41.378664 (+ 154us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:41.378712 (+ 48us) write_op.cc:312] APPLY: finished
0504 13:56:41.378906 (+ 194us) log.cc:844] Serialized 23 byte log entry
0504 13:56:41.379622 (+ 716us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:41.379690 (+ 68us) write_op.cc:454] Released schema lock
0504 13:56:41.379723 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0504 13:56:41.528930 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:41.529389 (+ 459us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:41.529418 (+ 29us) write_op.cc:435] Acquired schema lock
0504 13:56:41.529433 (+ 15us) tablet.cc:662] Decoding operations
0504 13:56:41.529718 (+ 285us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:41.529764 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:41.529787 (+ 23us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:41.529939 (+ 152us) tablet.cc:701] Row locks acquired
0504 13:56:41.529951 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:41.530064 (+ 113us) write_op.cc:270] Start()
0504 13:56:41.530626 (+ 562us) write_op.cc:276] Timestamp: P: 1777903001530014 usec, L: 0
0504 13:56:41.530642 (+ 16us) op_driver.cc:348] REPLICATION: starting
0504 13:56:41.531122 (+ 480us) log.cc:844] Serialized 356 byte log entry
0504 13:56:41.539086 (+ 7964us) op_driver.cc:464] REPLICATION: finished
0504 13:56:41.539481 (+ 395us) write_op.cc:301] APPLY: starting
0504 13:56:41.539545 (+ 64us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:41.539631 (+ 86us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:41.540210 (+ 579us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:41.540352 (+ 142us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:41.540480 (+ 128us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:41.540529 (+ 49us) write_op.cc:312] APPLY: finished
0504 13:56:41.540676 (+ 147us) log.cc:844] Serialized 23 byte log entry
0504 13:56:41.540836 (+ 160us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:41.540916 (+ 80us) write_op.cc:454] Released schema lock
0504 13:56:41.540944 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0504 13:56:41.671287 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:41.671740 (+ 453us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:41.671766 (+ 26us) write_op.cc:435] Acquired schema lock
0504 13:56:41.671782 (+ 16us) tablet.cc:662] Decoding operations
0504 13:56:41.672043 (+ 261us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:41.672088 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:41.672113 (+ 25us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:41.672255 (+ 142us) tablet.cc:701] Row locks acquired
0504 13:56:41.672267 (+ 12us) write_op.cc:260] PREPARE: finished
0504 13:56:41.672372 (+ 105us) write_op.cc:270] Start()
0504 13:56:41.672458 (+ 86us) write_op.cc:276] Timestamp: P: 1777903001672314 usec, L: 0
0504 13:56:41.672473 (+ 15us) op_driver.cc:348] REPLICATION: starting
0504 13:56:41.672966 (+ 493us) log.cc:844] Serialized 356 byte log entry
0504 13:56:41.679982 (+ 7016us) op_driver.cc:464] REPLICATION: finished
0504 13:56:41.680290 (+ 308us) write_op.cc:301] APPLY: starting
0504 13:56:41.680345 (+ 55us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:41.680425 (+ 80us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:41.680507 (+ 82us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:41.680681 (+ 174us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:41.680803 (+ 122us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:41.680852 (+ 49us) write_op.cc:312] APPLY: finished
0504 13:56:41.681006 (+ 154us) log.cc:844] Serialized 23 byte log entry
0504 13:56:41.681138 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:41.681198 (+ 60us) write_op.cc:454] Released schema lock
0504 13:56:41.681228 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0504 13:56:41.845918 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:41.846332 (+ 414us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:41.846356 (+ 24us) write_op.cc:435] Acquired schema lock
0504 13:56:41.846371 (+ 15us) tablet.cc:662] Decoding operations
0504 13:56:41.846670 (+ 299us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:41.846708 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:41.846733 (+ 25us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:41.846880 (+ 147us) tablet.cc:701] Row locks acquired
0504 13:56:41.846891 (+ 11us) write_op.cc:260] PREPARE: finished
0504 13:56:41.847008 (+ 117us) write_op.cc:270] Start()
0504 13:56:41.847093 (+ 85us) write_op.cc:276] Timestamp: P: 1777903001846933 usec, L: 0
0504 13:56:41.847112 (+ 19us) op_driver.cc:348] REPLICATION: starting
0504 13:56:41.847613 (+ 501us) log.cc:844] Serialized 356 byte log entry
0504 13:56:41.858507 (+ 10894us) spinlock_profiling.cc:243] Waited 7.93 ms on lock 0x7b5400020920. stack: 00007efe36c52c83 00007efe36c52b80 00007efe367c3574 000000000050e094 000000000050e029 00007efe3dcab470 00007efe3dcb8a98 00007efe3dcb86da 00007efe3dd3cd45 00007efe3dd3d1e9 00007efe3e1af9f1 00007efe3e1af1c8 00007efe3e1b3e61 00007efe3e1b3e19 00007efe3e1b3da9 00007efe3e1b3d71
0504 13:56:41.859972 (+ 1465us) op_driver.cc:464] REPLICATION: finished
0504 13:56:41.861597 (+ 1625us) write_op.cc:301] APPLY: starting
0504 13:56:41.861645 (+ 48us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:41.861739 (+ 94us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:41.861757 (+ 18us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:41.861888 (+ 131us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:41.861995 (+ 107us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:41.862045 (+ 50us) write_op.cc:312] APPLY: finished
0504 13:56:41.862187 (+ 142us) log.cc:844] Serialized 23 byte log entry
0504 13:56:41.862309 (+ 122us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:41.862365 (+ 56us) write_op.cc:454] Released schema lock
0504 13:56:41.862392 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:41.863515 (+ 1123us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:41.863539 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":77014,"reported_tablets":99,"child_traces":[["op",{"apply.queue_time_us":36632,"apply.run_cpu_time_us":3062,"apply.run_wall_time_us":166124,"num_ops":11,"prepare.queue_time_us":33215,"prepare.run_cpu_time_us":4487,"prepare.run_wall_time_us":63569,"raft.queue_time_us":296,"raft.run_cpu_time_us":708,"raft.run_wall_time_us":709,"replication_time_us":71084,"thread_start_us":97545,"threads_started":3,"wal-append.queue_time_us":6575,"wal-append.run_cpu_time_us":15349,"wal-append.run_wall_time_us":1735675}],["op",{"apply.queue_time_us":14199,"apply.run_cpu_time_us":1324,"apply.run_wall_time_us":1979,"num_ops":1,"prepare.queue_time_us":101,"prepare.run_cpu_time_us":2021,"prepare.run_wall_time_us":2997,"raft.queue_time_us":196,"raft.run_cpu_time_us":696,"raft.run_wall_time_us":24294,"replication_time_us":21268,"thread_start_us":42449,"threads_started":1}],["op",{"apply.queue_time_us":259,"apply.run_cpu_time_us":1627,"apply.run_wall_time_us":1627,"num_ops":1,"prepare.queue_time_us":1387,"prepare.run_cpu_time_us":2501,"prepare.run_wall_time_us":2501,"raft.queue_time_us":1074,"raft.run_cpu_time_us":559,"raft.run_wall_time_us":558,"replication_time_us":8360}],["op",{"apply.queue_time_us":172,"apply.run_cpu_time_us":1096,"apply.run_wall_time_us":1095,"num_ops":1,"prepare.queue_time_us":93,"prepare.run_cpu_time_us":1955,"prepare.run_wall_time_us":1956,"raft.queue_time_us":1085,"raft.run_cpu_time_us":596,"raft.run_wall_time_us":596,"replication_time_us":7411}],["op",{"apply.queue_time_us":1509,"apply.run_cpu_time_us":947,"apply.run_wall_time_us":947,"num_ops":1,"prepare.queue_time_us":1923,"prepare.run_cpu_time_us":3879,"prepare.run_wall_time_us":17585,"raft.queue_time_us":310,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":190,"replication_time_us":12782,"spinlock_wait_cycles":18239360}]]}
W20260504 13:56:51.399351 9727 debug-util.cc:398] Leaking SignalData structure 0x7b080015bfa0 after lost signal to thread 9597
W20260504 13:56:51.400444 9727 debug-util.cc:398] Leaking SignalData structure 0x7b0800176860 after lost signal to thread 9730
I20260504 13:56:51.419030 11826 ts_tablet_manager.cc:1929] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:51.420198 11826 log.cc:1199] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/21d6358b451d4b8d87a22e30e0780f44
I20260504 13:56:51.421043 11826 ts_tablet_manager.cc:1950] T 21d6358b451d4b8d87a22e30e0780f44 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:51.423641 11826 tablet_replica.cc:333] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:51.424525 11826 raft_consensus.cc:2243] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:51.425102 11826 raft_consensus.cc:2272] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:51.427016 11826 ts_tablet_manager.cc:1916] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:51.428680 11711 consensus_queue.cc:1048] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:51.436612 9832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7989338bc4bf4bad8fb8fe37587d0a16" candidate_uuid: "7b79fe7dd8934dd996393a2d96eabf00" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4051d2bf0bf410eb8acd107560df888" is_pre_election: true
I20260504 13:56:51.442622 11843 tablet.cc:1724] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:51.449062 9173 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 8e4e583afc804212bece2f727411cc14 (table default.Table_a8bc83249ad949c49e759a07818e2c02 [id=4182fad7cb2344d88498c26192164a10]) successfully deleted
I20260504 13:56:51.449815 9173 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet 21d6358b451d4b8d87a22e30e0780f44 (table default.Table_5bfac12ab2f84c06b1f992848d8f5041 [id=c2f8dc1bfbc14f8a8c093490ff05a18c]) successfully deleted
I20260504 13:56:51.469663 11826 ts_tablet_manager.cc:1929] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:51.471509 11826 log.cc:1199] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/da61c91e10264fa49fe397d9a9b014bb
I20260504 13:56:51.472350 11826 ts_tablet_manager.cc:1950] T da61c91e10264fa49fe397d9a9b014bb P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:51.488452 9187 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:45802 (request call id 34) took 3601 ms. Trace:
I20260504 13:56:51.488793 9187 rpcz_store.cc:276] 0504 13:56:47.886857 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:51.487845 (+3600988us) service_pool.cc:224] Handling call
0504 13:56:51.488387 (+ 542us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:51.496939 9181 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: default.Table_9b40ba316993413181a9743fb7bab7fb [id=fb4e5ce5526e44de8effdd766f4c257f] alter complete (version 1)
I20260504 13:56:51.489321 9173 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet da61c91e10264fa49fe397d9a9b014bb (table default.Table_999e37d0dab54ca68c0fa66f74582109 [id=e37fd7e36e604a60832acf303f0ecb7f]) successfully deleted
I20260504 13:56:51.591109 11843 tablet.cc:1724] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:51.593330 11849 consensus_queue.cc:1048] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260504 13:56:51.602010 9183 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:45997 (request call id 271) took 8233 ms (client timeout 1000 ms). Trace:
I20260504 13:56:51.605885 9659 tablet_service.cc:1558] Processing DeleteTablet for tablet da61c91e10264fa49fe397d9a9b014bb with delete_type TABLET_DATA_DELETED (replaced by tablet 6323e1c10e7643f8a28f04e285f620e9) from {username='slave'} at 127.0.0.1:34768
W20260504 13:56:51.602403 9183 rpcz_store.cc:269] 0504 13:56:43.368578 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:43.368779 (+ 201us) service_pool.cc:224] Handling call
0504 13:56:51.601911 (+8233132us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:47.193200 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:47.194341 (+ 1141us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:47.194378 (+ 37us) write_op.cc:435] Acquired schema lock
0504 13:56:47.194400 (+ 22us) tablet.cc:662] Decoding operations
0504 13:56:47.195057 (+ 657us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:47.195125 (+ 68us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:47.195151 (+ 26us) tablet.cc:685] Acquiring locks for 5 operations
0504 13:56:47.195708 (+ 557us) tablet.cc:701] Row locks acquired
0504 13:56:47.195721 (+ 13us) write_op.cc:260] PREPARE: finished
0504 13:56:47.195869 (+ 148us) write_op.cc:270] Start()
0504 13:56:47.195970 (+ 101us) write_op.cc:276] Timestamp: P: 1777903007195809 usec, L: 0
0504 13:56:47.195998 (+ 28us) op_driver.cc:348] REPLICATION: starting
0504 13:56:47.196577 (+ 579us) log.cc:844] Serialized 2054 byte log entry
0504 13:56:47.390308 (+193731us) op_driver.cc:464] REPLICATION: finished
0504 13:56:47.439259 (+ 48951us) write_op.cc:301] APPLY: starting
0504 13:56:47.439325 (+ 66us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:47.439502 (+ 177us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:47.439645 (+ 143us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:47.440849 (+ 1204us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:47.441051 (+ 202us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0504 13:56:47.441103 (+ 52us) write_op.cc:312] APPLY: finished
0504 13:56:47.441348 (+ 245us) log.cc:844] Serialized 47 byte log entry
0504 13:56:47.443483 (+ 2135us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:47.443565 (+ 82us) write_op.cc:454] Released schema lock
0504 13:56:47.475260 (+ 31695us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:47.475579 (+ 319us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:47.475607 (+ 28us) write_op.cc:454] Released schema lock
Related trace 'op':
0504 13:56:47.497058 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:47.497515 (+ 457us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:47.497543 (+ 28us) write_op.cc:435] Acquired schema lock
0504 13:56:47.497557 (+ 14us) tablet.cc:662] Decoding operations
0504 13:56:47.497880 (+ 323us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:47.497925 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:47.497956 (+ 31us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:47.498118 (+ 162us) tablet.cc:701] Row locks acquired
0504 13:56:47.498134 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:47.498251 (+ 117us) write_op.cc:270] Start()
0504 13:56:47.498348 (+ 97us) write_op.cc:276] Timestamp: P: 1777903007498175 usec, L: 0
0504 13:56:47.498367 (+ 19us) op_driver.cc:348] REPLICATION: starting
0504 13:56:47.498890 (+ 523us) log.cc:844] Serialized 356 byte log entry
0504 13:56:47.507561 (+ 8671us) op_driver.cc:464] REPLICATION: finished
0504 13:56:47.513399 (+ 5838us) write_op.cc:301] APPLY: starting
0504 13:56:47.513456 (+ 57us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:47.513540 (+ 84us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:47.513620 (+ 80us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:47.513850 (+ 230us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:47.513988 (+ 138us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:47.514033 (+ 45us) write_op.cc:312] APPLY: finished
0504 13:56:47.514212 (+ 179us) log.cc:844] Serialized 23 byte log entry
0504 13:56:47.514865 (+ 653us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:47.514934 (+ 69us) write_op.cc:454] Released schema lock
0504 13:56:47.514972 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:47.515317 (+ 345us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:47.515343 (+ 26us) write_op.cc:454] Released schema lock
Related trace 'op':
0504 13:56:51.007385 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:51.007834 (+ 449us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:51.007865 (+ 31us) write_op.cc:435] Acquired schema lock
0504 13:56:51.007882 (+ 17us) tablet.cc:662] Decoding operations
0504 13:56:51.008147 (+ 265us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:51.008182 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:51.008206 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:51.008351 (+ 145us) tablet.cc:701] Row locks acquired
0504 13:56:51.008361 (+ 10us) write_op.cc:260] PREPARE: finished
0504 13:56:51.008482 (+ 121us) write_op.cc:270] Start()
0504 13:56:51.008565 (+ 83us) write_op.cc:276] Timestamp: P: 1777903011008404 usec, L: 0
0504 13:56:51.008583 (+ 18us) op_driver.cc:348] REPLICATION: starting
0504 13:56:51.009017 (+ 434us) log.cc:844] Serialized 356 byte log entry
0504 13:56:51.064663 (+ 55646us) op_driver.cc:464] REPLICATION: finished
0504 13:56:51.064917 (+ 254us) write_op.cc:301] APPLY: starting
0504 13:56:51.064973 (+ 56us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:51.065063 (+ 90us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:51.065165 (+ 102us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:51.065356 (+ 191us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:51.065474 (+ 118us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:51.065518 (+ 44us) write_op.cc:312] APPLY: finished
0504 13:56:51.065695 (+ 177us) log.cc:844] Serialized 23 byte log entry
0504 13:56:51.065836 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:51.065890 (+ 54us) write_op.cc:454] Released schema lock
0504 13:56:51.065922 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":74062,"reported_tablets":100,"child_traces":[["op",{"apply.queue_time_us":48726,"apply.run_cpu_time_us":2738,"apply.run_wall_time_us":36307,"num_ops":5,"prepare.queue_time_us":68769,"prepare.run_cpu_time_us":3977,"prepare.run_wall_time_us":110623,"replication_time_us":194209,"spinlock_wait_cycles":220800,"thread_start_us":173071,"threads_started":3,"wal-append.queue_time_us":68172}],["op",{"apply.queue_time_us":5695,"apply.run_cpu_time_us":1287,"apply.run_wall_time_us":1885,"num_ops":1,"prepare.queue_time_us":290,"prepare.run_cpu_time_us":2109,"prepare.run_wall_time_us":2109,"raft.queue_time_us":1116,"raft.run_cpu_time_us":632,"raft.run_wall_time_us":631,"replication_time_us":9108}],["op",{"apply.queue_time_us":109,"apply.run_cpu_time_us":1199,"apply.run_wall_time_us":1200,"num_ops":1,"prepare.queue_time_us":2118,"prepare.run_cpu_time_us":1849,"prepare.run_wall_time_us":1850,"replication_time_us":55983}]]}
W20260504 13:56:51.628652 9185 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:45997 (request call id 270) took 9589 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.628983 9185 rpcz_store.cc:269] 0504 13:56:42.039013 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:42.039159 (+ 146us) service_pool.cc:224] Handling call
0504 13:56:51.628575 (+9589416us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":236283,"reported_tablets":100}
W20260504 13:56:51.663484 9181 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:33659 (request call id 244) took 7342 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.665530 9181 rpcz_store.cc:269] 0504 13:56:44.321570 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:44.321766 (+ 196us) service_pool.cc:224] Handling call
0504 13:56:51.496618 (+7174852us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:51.496643 (+ 25us) write_op.cc:454] Released schema lock
0504 13:56:51.663414 (+166771us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:51.463300 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:51.463767 (+ 467us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:51.463794 (+ 27us) write_op.cc:435] Acquired schema lock
0504 13:56:51.463809 (+ 15us) tablet.cc:662] Decoding operations
0504 13:56:51.464115 (+ 306us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:51.464158 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:51.464180 (+ 22us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:51.464339 (+ 159us) tablet.cc:701] Row locks acquired
0504 13:56:51.464355 (+ 16us) write_op.cc:260] PREPARE: finished
0504 13:56:51.464476 (+ 121us) write_op.cc:270] Start()
0504 13:56:51.464561 (+ 85us) write_op.cc:276] Timestamp: P: 1777903011464411 usec, L: 0
0504 13:56:51.464575 (+ 14us) op_driver.cc:348] REPLICATION: starting
0504 13:56:51.465048 (+ 473us) log.cc:844] Serialized 356 byte log entry
0504 13:56:51.475290 (+ 10242us) op_driver.cc:464] REPLICATION: finished
0504 13:56:51.475620 (+ 330us) write_op.cc:301] APPLY: starting
0504 13:56:51.475670 (+ 50us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:51.475776 (+ 106us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:51.475863 (+ 87us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:51.476033 (+ 170us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:51.476187 (+ 154us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:51.476230 (+ 43us) write_op.cc:312] APPLY: finished
0504 13:56:51.476400 (+ 170us) log.cc:844] Serialized 23 byte log entry
0504 13:56:51.476537 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:51.476600 (+ 63us) write_op.cc:454] Released schema lock
0504 13:56:51.476629 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":142036,"reported_tablets":104,"child_traces":[["op",{"apply.queue_time_us":161,"apply.run_cpu_time_us":1182,"apply.run_wall_time_us":1181,"num_ops":1,"prepare.queue_time_us":10975,"prepare.run_cpu_time_us":2139,"prepare.run_wall_time_us":2139,"raft.queue_time_us":3205,"raft.run_cpu_time_us":1173,"raft.run_wall_time_us":1172,"replication_time_us":10624}]]}
W20260504 13:56:51.672343 9188 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.195:33659 (request call id 243) took 8842 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.679678 9188 rpcz_store.cc:269] 0504 13:56:42.830333 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:42.830483 (+ 150us) service_pool.cc:224] Handling call
0504 13:56:51.672284 (+8841801us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:48.474409 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:48.474922 (+ 513us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:48.474946 (+ 24us) write_op.cc:435] Acquired schema lock
0504 13:56:48.474961 (+ 15us) tablet.cc:662] Decoding operations
0504 13:56:48.483731 (+ 8770us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:48.483796 (+ 65us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:48.483824 (+ 28us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:48.484125 (+ 301us) tablet.cc:701] Row locks acquired
0504 13:56:48.484142 (+ 17us) write_op.cc:260] PREPARE: finished
0504 13:56:48.484287 (+ 145us) write_op.cc:270] Start()
0504 13:56:48.484376 (+ 89us) write_op.cc:276] Timestamp: P: 1777903008484211 usec, L: 0
0504 13:56:48.484394 (+ 18us) op_driver.cc:348] REPLICATION: starting
0504 13:56:48.484931 (+ 537us) log.cc:844] Serialized 913 byte log entry
0504 13:56:48.653475 (+168544us) op_driver.cc:464] REPLICATION: finished
0504 13:56:48.693064 (+ 39589us) write_op.cc:301] APPLY: starting
0504 13:56:48.693125 (+ 61us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:48.693233 (+ 108us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:48.693354 (+ 121us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:48.693629 (+ 275us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:48.693772 (+ 143us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:48.693814 (+ 42us) write_op.cc:312] APPLY: finished
0504 13:56:48.693984 (+ 170us) log.cc:844] Serialized 29 byte log entry
0504 13:56:48.694131 (+ 147us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:48.694196 (+ 65us) write_op.cc:454] Released schema lock
0504 13:56:48.694224 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:48.709149 (+ 14925us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:48.709175 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":280837,"reported_tablets":104,"child_traces":[["op",{"apply.queue_time_us":34893,"apply.run_cpu_time_us":1443,"apply.run_wall_time_us":16035,"mutex_wait_us":1188,"num_ops":2,"prepare.queue_time_us":100344,"prepare.run_cpu_time_us":2764,"prepare.run_wall_time_us":26379,"raft.queue_time_us":1850,"raft.run_cpu_time_us":1510,"raft.run_wall_time_us":1510,"replication_time_us":168942,"thread_start_us":140107,"threads_started":2}]]}
I20260504 13:56:51.694567 9505 tablet_service.cc:1558] Processing DeleteTablet for tablet 8e4e583afc804212bece2f727411cc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:23 UTC) from {username='slave'} at 127.0.0.1:40114
W20260504 13:56:51.673291 9180 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:45997 (request call id 272) took 7058 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.695605 9180 rpcz_store.cc:269] 0504 13:56:44.614776 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:44.614960 (+ 184us) service_pool.cc:224] Handling call
0504 13:56:51.673235 (+7058275us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:51.050181 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:51.050637 (+ 456us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:51.050662 (+ 25us) write_op.cc:435] Acquired schema lock
0504 13:56:51.050677 (+ 15us) tablet.cc:662] Decoding operations
0504 13:56:51.050988 (+ 311us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:51.051024 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:51.051048 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:51.051229 (+ 181us) tablet.cc:701] Row locks acquired
0504 13:56:51.051247 (+ 18us) write_op.cc:260] PREPARE: finished
0504 13:56:51.051359 (+ 112us) write_op.cc:270] Start()
0504 13:56:51.051455 (+ 96us) write_op.cc:276] Timestamp: P: 1777903011051278 usec, L: 0
0504 13:56:51.051474 (+ 19us) op_driver.cc:348] REPLICATION: starting
0504 13:56:51.055506 (+ 4032us) log.cc:844] Serialized 533 byte log entry
0504 13:56:51.060781 (+ 5275us) spinlock_profiling.cc:243] Waited 3.33 ms on lock 0x7b54000203d4. stack: 00007efe36c52c83 00007efe36c52b80 00007efe367c3574 000000000050e094 000000000050e029 00007efe3dcaa951 00007efe3dd3d208 00007efe3e1af9f1 00007efe3e1af1c8 00007efe3e1b3e61 00007efe3e1b3e19 00007efe3e1b3da9 00007efe3e1b3d71 00007efe3e1b2f9d 00000000005b3bc4 00000000005b39f9
0504 13:56:51.078614 (+ 17833us) op_driver.cc:464] REPLICATION: finished
0504 13:56:51.079232 (+ 618us) write_op.cc:301] APPLY: starting
0504 13:56:51.079283 (+ 51us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:51.079364 (+ 81us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:51.079382 (+ 18us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:51.079549 (+ 167us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:51.079664 (+ 115us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:51.079703 (+ 39us) write_op.cc:312] APPLY: finished
0504 13:56:51.079862 (+ 159us) log.cc:844] Serialized 23 byte log entry
0504 13:56:51.079987 (+ 125us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:51.080041 (+ 54us) write_op.cc:454] Released schema lock
0504 13:56:51.080069 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:51.080267 (+ 198us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:51.080290 (+ 23us) write_op.cc:454] Released schema lock
0504 13:56:51.097310 (+ 17020us) spinlock_profiling.cc:243] Waited 4.4 ms on lock 0x7b1800058ba0. stack: 00007efe36c52c83 00007efe36c52b80 00007efe367c3574 000000000050e094 000000000050e029 00007efe3fa7b251 00007efe3fa87ab1 00007efe36c52be6 00007efe36c52b80 00007efe367c3574 000000000050e094 000000000050e029 00007efe3dcab470 00007efe3dc9b81c 00007efe3dca2133 00007efe3dca20b9
0504 13:56:51.113682 (+ 16372us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b580008b1d0. stack: 00007efe36c52c83 00007efe36c52b80 00007efe367c3574 000000000050e094 000000000050e029 00007efe3dcab470 00007efe3dc9b81c 00007efe3dca2133 00007efe3dca20b9 00007efe3dca2049 00007efe3dca2011 00007efe3dca11dd 00000000005b3bc4 00000000005b39f9 00007efe36c84ab5 00007efe36c86e01
Metrics: {"mutex_wait_us":81059,"reported_tablets":100,"child_traces":[["op",{"apply.queue_time_us":454,"apply.run_cpu_time_us":1009,"apply.run_wall_time_us":1008,"num_ops":1,"prepare.queue_time_us":89,"prepare.run_cpu_time_us":3267,"prepare.run_wall_time_us":15565,"raft.queue_time_us":3443,"raft.run_cpu_time_us":3134,"raft.run_wall_time_us":49394,"replication_time_us":27028,"spinlock_wait_cycles":22578176}]]}
W20260504 13:56:51.706367 9186 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:39697 (request call id 228) took 10360 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.735942 9173 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 8e4e583afc804212bece2f727411cc14 because the tablet was not found. No further retry: Not found: Tablet not found: 8e4e583afc804212bece2f727411cc14
W20260504 13:56:51.706715 9186 rpcz_store.cc:269] 0504 13:56:41.346073 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:41.346164 (+ 91us) service_pool.cc:224] Handling call
0504 13:56:51.706251 (+10360087us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:43.708148 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:43.710915 (+ 2767us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:43.710954 (+ 39us) write_op.cc:435] Acquired schema lock
0504 13:56:43.711022 (+ 68us) tablet.cc:662] Decoding operations
0504 13:56:43.712250 (+ 1228us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:43.712340 (+ 90us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:43.712397 (+ 57us) tablet.cc:685] Acquiring locks for 3 operations
0504 13:56:43.712898 (+ 501us) tablet.cc:701] Row locks acquired
0504 13:56:43.712946 (+ 48us) write_op.cc:260] PREPARE: finished
0504 13:56:43.713145 (+ 199us) write_op.cc:270] Start()
0504 13:56:43.713293 (+ 148us) write_op.cc:276] Timestamp: P: 1777903003713060 usec, L: 0
0504 13:56:43.713338 (+ 45us) op_driver.cc:348] REPLICATION: starting
0504 13:56:43.797090 (+ 83752us) log.cc:844] Serialized 1293 byte log entry
0504 13:56:44.017469 (+220379us) op_driver.cc:464] REPLICATION: finished
0504 13:56:44.058933 (+ 41464us) write_op.cc:301] APPLY: starting
0504 13:56:44.059053 (+ 120us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:44.059240 (+ 187us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:44.059428 (+ 188us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:44.060050 (+ 622us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:44.060271 (+ 221us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0504 13:56:44.060343 (+ 72us) write_op.cc:312] APPLY: finished
0504 13:56:44.060766 (+ 423us) log.cc:844] Serialized 35 byte log entry
0504 13:56:44.060992 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:44.061143 (+ 151us) write_op.cc:454] Released schema lock
0504 13:56:44.061215 (+ 72us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:44.061716 (+ 501us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:44.061764 (+ 48us) write_op.cc:454] Released schema lock
Related trace 'op':
0504 13:56:51.140495 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:51.140945 (+ 450us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:51.140984 (+ 39us) write_op.cc:435] Acquired schema lock
0504 13:56:51.141000 (+ 16us) tablet.cc:662] Decoding operations
0504 13:56:51.141344 (+ 344us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:51.141391 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:51.141419 (+ 28us) tablet.cc:685] Acquiring locks for 1 operations
0504 13:56:51.141569 (+ 150us) tablet.cc:701] Row locks acquired
0504 13:56:51.141584 (+ 15us) write_op.cc:260] PREPARE: finished
0504 13:56:51.141704 (+ 120us) write_op.cc:270] Start()
0504 13:56:51.141952 (+ 248us) write_op.cc:276] Timestamp: P: 1777903011141608 usec, L: 0
0504 13:56:51.141972 (+ 20us) op_driver.cc:348] REPLICATION: starting
0504 13:56:51.142476 (+ 504us) log.cc:844] Serialized 356 byte log entry
0504 13:56:51.202974 (+ 60498us) op_driver.cc:464] REPLICATION: finished
0504 13:56:51.203236 (+ 262us) write_op.cc:301] APPLY: starting
0504 13:56:51.203298 (+ 62us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:51.203418 (+ 120us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:51.203539 (+ 121us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:51.203726 (+ 187us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:51.203905 (+ 179us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0504 13:56:51.203963 (+ 58us) write_op.cc:312] APPLY: finished
0504 13:56:51.204151 (+ 188us) log.cc:844] Serialized 23 byte log entry
0504 13:56:51.204470 (+ 319us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:51.204542 (+ 72us) write_op.cc:454] Released schema lock
0504 13:56:51.204588 (+ 46us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":377813,"reported_tablets":102,"child_traces":[["op",{"apply.queue_time_us":23261,"apply.run_cpu_time_us":2752,"apply.run_wall_time_us":20625,"num_ops":3,"prepare.queue_time_us":94902,"prepare.run_cpu_time_us":4040,"prepare.run_wall_time_us":136649,"raft.queue_time_us":122899,"raft.run_cpu_time_us":11383,"raft.run_wall_time_us":25984,"replication_time_us":303964,"spinlock_wait_cycles":19840,"thread_start_us":168167,"threads_started":3,"wal-append.queue_time_us":96033,"wal-append.run_cpu_time_us":2739,"wal-append.run_wall_time_us":1182685}],["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":1474,"apply.run_wall_time_us":1678,"num_ops":1,"prepare.queue_time_us":22407,"prepare.run_cpu_time_us":2087,"prepare.run_wall_time_us":50651,"replication_time_us":60895}]]}
I20260504 13:56:51.745904 9505 tablet_service.cc:1558] Processing DeleteTablet for tablet 21d6358b451d4b8d87a22e30e0780f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:25 UTC) from {username='slave'} at 127.0.0.1:40114
I20260504 13:56:51.746611 9507 tablet_service.cc:1558] Processing DeleteTablet for tablet da61c91e10264fa49fe397d9a9b014bb with delete_type TABLET_DATA_DELETED (replaced by tablet 6323e1c10e7643f8a28f04e285f620e9) from {username='slave'} at 127.0.0.1:40114
W20260504 13:56:51.766978 9173 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 21d6358b451d4b8d87a22e30e0780f44 because the tablet was not found. No further retry: Not found: Tablet not found: 21d6358b451d4b8d87a22e30e0780f44
W20260504 13:56:51.769265 9173 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet da61c91e10264fa49fe397d9a9b014bb because the tablet was not found. No further retry: Not found: Tablet not found: da61c91e10264fa49fe397d9a9b014bb
I20260504 13:56:51.793922 11818 ts_tablet_manager.cc:1929] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:51.794374 11818 log.cc:1199] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/da61c91e10264fa49fe397d9a9b014bb
I20260504 13:56:51.796988 11818 ts_tablet_manager.cc:1950] T da61c91e10264fa49fe397d9a9b014bb P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:51.806671 11856 tablet_replica.cc:333] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:51.814881 11818 tablet_replica.cc:333] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:51.817759 11818 raft_consensus.cc:2243] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:51.818308 11818 raft_consensus.cc:2272] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:51.821568 11818 ts_tablet_manager.cc:1916] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:51.825248 9175 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet da61c91e10264fa49fe397d9a9b014bb (table default.Table_999e37d0dab54ca68c0fa66f74582109 [id=e37fd7e36e604a60832acf303f0ecb7f]) successfully deleted
W20260504 13:56:51.880955 9618 consensus_peers.cc:597] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da61c91e10264fa49fe397d9a9b014bb. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:51.889209 11843 tablet.cc:1724] T 5cb30835150949f39f6e7e7563381763 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:51.890739 11818 ts_tablet_manager.cc:1929] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:51.891148 11818 log.cc:1199] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/8bb77106b48a45a2aaa938dc474fa8bd
W20260504 13:56:51.900954 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 57448.4us
I20260504 13:56:51.903765 11818 ts_tablet_manager.cc:1950] T 8bb77106b48a45a2aaa938dc474fa8bd P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:51.905573 9175 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 8bb77106b48a45a2aaa938dc474fa8bd (table default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]) successfully deleted
I20260504 13:56:51.904901 11818 tablet_replica.cc:333] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:51.911892 11818 raft_consensus.cc:2243] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:51.912431 11818 raft_consensus.cc:2272] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:51.919684 11818 ts_tablet_manager.cc:1916] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:51.942072 11832 tablet.cc:1724] T 2ddf890c17964e7f90409ebe3c4034ef P 5beceb6317444247b31b44a6581707c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:51.943733 9507 tablet_service.cc:1558] Processing DeleteTablet for tablet 8e4e583afc804212bece2f727411cc14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:23 UTC) from {username='slave'} at 127.0.0.1:40114
I20260504 13:56:51.944821 9507 tablet_service.cc:1558] Processing DeleteTablet for tablet 21d6358b451d4b8d87a22e30e0780f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:25 UTC) from {username='slave'} at 127.0.0.1:40114
I20260504 13:56:51.966733 11838 consensus_queue.cc:1048] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:51.976449 9507 tablet_service.cc:1558] Processing DeleteTablet for tablet da61c91e10264fa49fe397d9a9b014bb with delete_type TABLET_DATA_DELETED (replaced by tablet 6323e1c10e7643f8a28f04e285f620e9) from {username='slave'} at 127.0.0.1:40114
W20260504 13:56:51.979786 9182 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.8.238.193:39697 (request call id 229) took 8295 ms (client timeout 1000 ms). Trace:
W20260504 13:56:51.980099 9182 rpcz_store.cc:269] 0504 13:56:43.684561 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:43.684738 (+ 177us) service_pool.cc:224] Handling call
0504 13:56:43.845795 (+161057us) spinlock_profiling.cc:243] Waited 82.1 ms on lock 0x7b54000203d4. stack: 00007efe36c52c83 00007efe36c52b80 00007efe367c3574 000000000050e094 000000000050e029 00007efe3dcab470 00007efe3dd4a31a 00007efe3fa5e46c 00007efe3fb583ee 00007efe3d75f4a4 00007efe3d75f432 00007efe3d75f361 00007efe3d75f2dc 00007efe3d75e592 00007efe3ba34bcc 00007efe3ba34006
0504 13:56:51.979725 (+8133930us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0504 13:56:49.367251 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0504 13:56:49.367728 (+ 477us) write_op.cc:432] Acquiring schema lock in shared mode
0504 13:56:49.367757 (+ 29us) write_op.cc:435] Acquired schema lock
0504 13:56:49.367773 (+ 16us) tablet.cc:662] Decoding operations
0504 13:56:49.368941 (+ 1168us) write_op.cc:620] Acquiring the partition lock for write op
0504 13:56:49.368995 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0504 13:56:49.369020 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0504 13:56:49.369287 (+ 267us) tablet.cc:701] Row locks acquired
0504 13:56:49.369304 (+ 17us) write_op.cc:260] PREPARE: finished
0504 13:56:49.369429 (+ 125us) write_op.cc:270] Start()
0504 13:56:49.369518 (+ 89us) write_op.cc:276] Timestamp: P: 1777903009369374 usec, L: 0
0504 13:56:49.369536 (+ 18us) op_driver.cc:348] REPLICATION: starting
0504 13:56:49.370105 (+ 569us) log.cc:844] Serialized 913 byte log entry
0504 13:56:49.430864 (+ 60759us) op_driver.cc:464] REPLICATION: finished
0504 13:56:49.448213 (+ 17349us) write_op.cc:301] APPLY: starting
0504 13:56:49.448270 (+ 57us) tablet.cc:1367] starting BulkCheckPresence
0504 13:56:49.448367 (+ 97us) tablet.cc:1370] finished BulkCheckPresence
0504 13:56:49.448480 (+ 113us) tablet.cc:1372] starting ApplyRowOperation cycle
0504 13:56:49.448792 (+ 312us) tablet.cc:1383] finished ApplyRowOperation cycle
0504 13:56:49.452427 (+ 3635us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0504 13:56:49.452473 (+ 46us) write_op.cc:312] APPLY: finished
0504 13:56:49.455369 (+ 2896us) log.cc:844] Serialized 29 byte log entry
0504 13:56:49.456079 (+ 710us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:49.456145 (+ 66us) write_op.cc:454] Released schema lock
0504 13:56:49.456176 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0504 13:56:49.456421 (+ 245us) write_op.cc:489] Releasing partition, row and schema locks
0504 13:56:49.456451 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":196211,"reported_tablets":102,"spinlock_wait_cycles":188915456,"child_traces":[["op",{"apply.queue_time_us":17180,"apply.run_cpu_time_us":1471,"apply.run_wall_time_us":8184,"num_ops":2,"prepare.queue_time_us":9430,"prepare.run_cpu_time_us":2571,"prepare.run_wall_time_us":18249,"raft.queue_time_us":2324,"raft.run_cpu_time_us":1483,"raft.run_wall_time_us":1482,"replication_time_us":61224,"thread_start_us":37622,"threads_started":2}]]}
W20260504 13:56:51.986706 9173 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 8e4e583afc804212bece2f727411cc14 because the tablet was not found. No further retry: Not found: Tablet not found: 8e4e583afc804212bece2f727411cc14
I20260504 13:56:51.989318 11818 ts_tablet_manager.cc:1929] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:51.989732 11818 log.cc:1199] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/8e4e583afc804212bece2f727411cc14
I20260504 13:56:51.990463 11818 ts_tablet_manager.cc:1950] T 8e4e583afc804212bece2f727411cc14 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:51.993672 11856 raft_consensus.cc:2243] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:51.994567 11856 raft_consensus.cc:2272] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:51.996661 11856 ts_tablet_manager.cc:1916] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260504 13:56:52.000939 9173 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet 21d6358b451d4b8d87a22e30e0780f44 because the tablet was not found. No further retry: Not found: Tablet not found: 21d6358b451d4b8d87a22e30e0780f44
W20260504 13:56:52.001508 9173 catalog_manager.cc:4969] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): delete failed for tablet da61c91e10264fa49fe397d9a9b014bb because the tablet was not found. No further retry: Not found: Tablet not found: da61c91e10264fa49fe397d9a9b014bb
W20260504 13:56:52.048678 9767 consensus_peers.cc:597] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21d6358b451d4b8d87a22e30e0780f44. This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:52.051050 11854 consensus_queue.cc:1048] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:56:52.061386 11818 tablet_replica.cc:333] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:52.062348 11818 raft_consensus.cc:2243] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:52.062883 11818 raft_consensus.cc:2272] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:52.064925 11818 ts_tablet_manager.cc:1916] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:52.068827 9175 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 8e4e583afc804212bece2f727411cc14 (table default.Table_a8bc83249ad949c49e759a07818e2c02 [id=4182fad7cb2344d88498c26192164a10]) successfully deleted
I20260504 13:56:52.069518 11781 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260504 13:56:52.073680 11854 consensus_queue.cc:1048] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:52.086923 11854 consensus_queue.cc:1048] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260504 13:56:52.112934 11714 consensus_queue.cc:1048] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:52.133606 11714 consensus_queue.cc:1048] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:52.137518 11818 ts_tablet_manager.cc:1929] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:56:52.137938 11818 log.cc:1199] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/41d41f8ba6dc4bb18116cb2b2dfabff0
I20260504 13:56:52.138628 11818 ts_tablet_manager.cc:1950] T 41d41f8ba6dc4bb18116cb2b2dfabff0 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:52.140717 11818 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40598 (request call id 13) took 1052 ms. Trace:
I20260504 13:56:52.141065 11818 rpcz_store.cc:276] 0504 13:56:51.088454 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:51.088945 (+ 491us) service_pool.cc:224] Handling call
0504 13:56:52.061211 (+972266us) ts_tablet_manager.cc:1151] Deleting tablet 41d41f8ba6dc4bb18116cb2b2dfabff0
0504 13:56:52.061260 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:52.096869 (+ 35609us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:52.137393 (+ 40524us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:52.140660 (+ 3267us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":971746}
W20260504 13:56:52.142923 9473 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260504 13:56:52.173463 11818 tablet_replica.cc:333] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:52.174587 11818 raft_consensus.cc:2243] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:52.175499 11818 raft_consensus.cc:2272] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:52.183863 11818 ts_tablet_manager.cc:1916] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:52.188052 9474 leader_election.cc:304] T 7989338bc4bf4bad8fb8fe37587d0a16 P 7b79fe7dd8934dd996393a2d96eabf00 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b79fe7dd8934dd996393a2d96eabf00; no voters: 5beceb6317444247b31b44a6581707c6, d4051d2bf0bf410eb8acd107560df888
I20260504 13:56:52.188776 11480 raft_consensus.cc:2749] T 7989338bc4bf4bad8fb8fe37587d0a16 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:56:52.142858 11856 ts_tablet_manager.cc:1929] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:52.191514 11856 log.cc:1199] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/da61c91e10264fa49fe397d9a9b014bb
I20260504 13:56:52.192565 11856 ts_tablet_manager.cc:1950] T da61c91e10264fa49fe397d9a9b014bb P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
W20260504 13:56:52.247812 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 44406.3us
W20260504 13:56:52.255782 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 62732.5us
W20260504 13:56:52.270957 9174 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92480654807d495b879576b0195871bd consensus state has no leader
I20260504 13:56:52.290130 11480 consensus_queue.cc:1048] T 564d0fafb49a4b4bb3bdb240c74c3410 P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:52.303978 11818 ts_tablet_manager.cc:1929] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:52.304419 11818 log.cc:1199] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/21d6358b451d4b8d87a22e30e0780f44
I20260504 13:56:52.320590 11818 ts_tablet_manager.cc:1950] T 21d6358b451d4b8d87a22e30e0780f44 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:52.346710 11870 consensus_queue.cc:1048] T bd635a863f7a41d087947f619e41f95c P 7b79fe7dd8934dd996393a2d96eabf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:56:52.369886 11818 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40598 (request call id 14) took 1281 ms. Trace:
I20260504 13:56:52.370241 11818 rpcz_store.cc:276] 0504 13:56:51.088733 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:51.089677 (+ 944us) service_pool.cc:224] Handling call
0504 13:56:52.171675 (+1081998us) ts_tablet_manager.cc:1151] Deleting tablet 21d6358b451d4b8d87a22e30e0780f44
0504 13:56:52.173324 (+ 1649us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0504 13:56:52.241031 (+ 67707us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:52.303803 (+ 62772us) tablet_metadata.cc:665] Metadata flushed
0504 13:56:52.369828 (+ 66025us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1081484}
I20260504 13:56:52.470523 11880 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 806407748d564439807fe5d73db9abda)
I20260504 13:56:52.471767 11880 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:52.539989 11881 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 806407748d564439807fe5d73db9abda)
I20260504 13:56:52.540723 11881 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:52.558413 11879 tablet.cc:1724] T c47b6e3b4d0c45c28384f7aae6b749de P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260504 13:56:52.566459 11880 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:52.566788 9352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:52.570504 11881 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:56:52.571336 10814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:52.580372 11884 tablet.cc:1724] T 2ddf890c17964e7f90409ebe3c4034ef P 7b79fe7dd8934dd996393a2d96eabf00: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260504 13:56:52.670449 11891 tablet.cc:1724] T a862a43c5ac443d9bac98e0472ca23b2 P d4051d2bf0bf410eb8acd107560df888: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20260504 13:56:52.948444 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 137524us
W20260504 13:56:53.061101 9331 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.238.254:36145 timed out after 0.516s (SENT)
W20260504 13:56:53.072129 9769 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260504 13:56:53.220849 10690 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.238.254:36145 timed out after 0.721s (SENT)
W20260504 13:56:53.400910 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 46324.9us
W20260504 13:56:53.606024 9331 leader_election.cc:336] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 806407748d564439807fe5d73db9abda (127.8.238.254:36145): Timed out: RequestConsensusVote RPC to 127.8.238.254:36145 timed out after 0.516s (SENT)
I20260504 13:56:53.606436 9331 leader_election.cc:304] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec9f4c3a43d84fc9a420dd10723c01a1; no voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda
W20260504 13:56:53.656806 9331 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.254:36145, user_credentials={real_user=slave}} blocked reactor thread for 50808.2us
I20260504 13:56:53.667621 11905 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260504 13:56:53.983634 10690 leader_election.cc:336] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 806407748d564439807fe5d73db9abda (127.8.238.254:36145): Timed out: RequestConsensusVote RPC to 127.8.238.254:36145 timed out after 0.721s (SENT)
I20260504 13:56:53.984117 10690 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:54.011890 11906 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 13:56:54.265976 11910 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 806407748d564439807fe5d73db9abda)
I20260504 13:56:54.266723 11910 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:54.270321 10814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:54.271919 11910 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:56:54.558980 11911 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 806407748d564439807fe5d73db9abda)
I20260504 13:56:54.559504 11911 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:54.562515 11911 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:54.563042 9352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
W20260504 13:56:55.145766 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 86695.5us
W20260504 13:56:55.220362 10690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.254:36145, user_credentials={real_user=slave}} blocked reactor thread for 64354.6us
W20260504 13:56:55.600904 9331 leader_election.cc:336] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 806407748d564439807fe5d73db9abda (127.8.238.254:36145): Timed out: RequestConsensusVote RPC to 127.8.238.254:36145 timed out after 0.654s (SENT)
I20260504 13:56:55.601318 9331 leader_election.cc:304] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec9f4c3a43d84fc9a420dd10723c01a1; no voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda
I20260504 13:56:55.603861 11920 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260504 13:56:55.740940 9175 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 57983fcecdb64a42884f797b4db57df0 consensus state has no leader
W20260504 13:56:55.816674 10690 leader_election.cc:336] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 806407748d564439807fe5d73db9abda (127.8.238.254:36145): Timed out: RequestConsensusVote RPC to 127.8.238.254:36145 timed out after 0.594s (SENT)
I20260504 13:56:55.817201 10690 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:55.836460 11921 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 13:56:56.591812 11927 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 806407748d564439807fe5d73db9abda)
I20260504 13:56:56.592280 11927 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:56.601035 10814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "6caedf407d13459e8fd477e8b60b7dc7" is_pre_election: true
I20260504 13:56:56.600021 11927 leader_election.cc:290] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), 6caedf407d13459e8fd477e8b60b7dc7 (127.8.238.253:36337)
I20260504 13:56:56.794045 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_2f0fc59ee89c402cb8a0fc23341636e6 [id=00b5922717934e15993bafde9f2513e7]
W20260504 13:56:56.821974 9620 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 61208.7us
I20260504 13:56:56.838635 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_798d2540b8f4489fb11763ccaa36bced [id=049b4fa786fc4dbb8e52931dcaaab90f]
I20260504 13:56:56.840822 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_512ae07c2df74cb99bb77006d985f2da [id=09959f81c69c49778a6ee33dc1e85aae]
I20260504 13:56:56.848184 11933 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 806407748d564439807fe5d73db9abda)
I20260504 13:56:56.850466 11933 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "806407748d564439807fe5d73db9abda" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 36145 } } peers { permanent_uuid: "6caedf407d13459e8fd477e8b60b7dc7" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 36337 } } peers { permanent_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 44781 } }
I20260504 13:56:56.924428 11933 leader_election.cc:290] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 806407748d564439807fe5d73db9abda (127.8.238.254:36145), ec9f4c3a43d84fc9a420dd10723c01a1 (127.8.238.252:44781)
I20260504 13:56:56.925482 9352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" is_pre_election: true
I20260504 13:56:56.944916 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38]
I20260504 13:56:56.946272 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_dd11ea89ad7248c39ec114af58cf598f [id=14bd1983ba95468e9e70d8f5724f42e6]
I20260504 13:56:56.983355 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_2bed5ea8f43149e8bc3b21cd9d1b3846 [id=15994ccc978545058e324152f70c8c3e]
I20260504 13:56:57.003855 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_dee1335eff89438e9c68835a4c258031 [id=1714df2b31ff4865b6b357500d121a26]
I20260504 13:56:57.031049 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_24398dd8ee63413ebb585587353ea155 [id=1d2bbdaf6382433cb8c2a55f62f0388d]
I20260504 13:56:57.056442 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_60fe6ee7a2414e2aa8ea2d57ee4a827d [id=1de44a1bbf06492084cfd70cf540b213]
I20260504 13:56:57.058539 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81]
I20260504 13:56:57.059849 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_5ff6dab0e47d41e681533fca02a36153 [id=230a8885dd2b4995815b650d3d9f9f75]
I20260504 13:56:57.061846 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_09255abca80f41dc895d81ccd89b9c7a [id=24485cf365ad4f9b89fa0f716fdb34cd]
I20260504 13:56:57.063108 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee]
W20260504 13:56:57.092944 9474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.195:34563, user_credentials={real_user=slave}} blocked reactor thread for 54105.4us
I20260504 13:56:57.131784 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_2585508fc4214c8b82d2957644309ac2 [id=256b605e5a2342ed883ec30e3de0ffda]
I20260504 13:56:57.137712 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_42f086bc327c42ccad96925c681a386a [id=2c6aa17153154189a33f5f3aa00b62e9]
W20260504 13:56:57.141029 9473 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.194:40369, user_credentials={real_user=slave}} blocked reactor thread for 58953.8us
I20260504 13:56:57.234390 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_69c9551ae2cf4ef399e40017a33bad73 [id=2d9b8dca85e9477eb77877d5618e9b6f]
I20260504 13:56:57.267714 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e]
I20260504 13:56:57.269966 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_1611aa9f4dba40e09dae4433e2d20154 [id=2fb2c56e2fb44b9aac476e06e36e5778]
I20260504 13:56:57.375933 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_2d5a582adb634895a55193dd861f3b60 [id=30f349197d564e8da82855fec17b4cd2]
I20260504 13:56:57.437453 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e]
I20260504 13:56:57.462083 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f]
I20260504 13:56:57.493818 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_963c93d5d8ed49e6a2052fa1e652da91 [id=399a65bb2fa6428296341de74e964518]
I20260504 13:56:57.522507 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_f8a23a8252f84cacb13b338f9d9dc9fc [id=39be2c4d16aa41c0832bfeb23269243a]
I20260504 13:56:57.573004 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_073a6dae9da444bea1f96b2f33fe1a85 [id=39c692c399dd43ca963357b6818871cf]
I20260504 13:56:57.644384 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_4207c9e8019c4088b934376c5568d101 [id=3ca89308de104a5fa27cac4c62047304]
I20260504 13:56:57.645769 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a]
W20260504 13:56:57.674515 10690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.254:36145, user_credentials={real_user=slave}} blocked reactor thread for 65237.3us
I20260504 13:56:57.647056 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_5944297f8c104978bbc6913ecd8f47a4 [id=3f11131cece44f3ebf3cd0f1e69139ae]
I20260504 13:56:57.712641 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12]
I20260504 13:56:57.714033 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_1665799e268a496d9ffc5568fa34a763 [id=403cbbc8a1304e5695a89d3697932c26]
I20260504 13:56:57.725675 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_003cbc6fdf0b4dddaba008af167dc621 [id=4629003a2d8d461c9e309d49ddecea6a]
I20260504 13:56:57.740032 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_a660204a2583425aa3ecd09c5cc01368 [id=4683a87561f14c049c1ed8eb68dc24d1]
I20260504 13:56:57.751608 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_821e9f466c974a2fbf24c07f8c266e62 [id=469bd5cb4e7f4a8a9c7af947a834ca7b]
I20260504 13:56:57.758425 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_f7b06aadc0024855b08b56cad8c7b6a3 [id=4efd9dd69847446d9f05700d3bd3f914]
I20260504 13:56:57.763406 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f]
I20260504 13:56:57.788062 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_2d928cb3553f40bc8e825d0741ef3199 [id=51b254be98ee457d892039784e4a5f0b]
I20260504 13:56:57.793921 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5]
I20260504 13:56:57.805159 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_af058f50341e4f478c970f8cb52bf0d7 [id=57ccd7218d454c8c908fe06b798bd83b]
I20260504 13:56:57.811435 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_597b4ca3e37849d0aa9c86eed64d0614 [id=5a597e1cae1d4126ae355289aef1ced6]
I20260504 13:56:57.820367 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62]
I20260504 13:56:57.821702 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_18f9a8030cea4cc0abedb81cd5accc59 [id=5c4f226dc33b428e92ca77959e075d7c]
I20260504 13:56:57.828019 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_2099388b6e2a455295be76a6d072758e [id=63e34a66cd644ab7bede27373e321714]
I20260504 13:56:57.829948 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_9bc232a8e4dd4df1a36b35d09dd50d0a [id=66faab34fb324213a874ea6fae3c96a1]
W20260504 13:56:57.830837 9767 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.193:33117, user_credentials={real_user=slave}} blocked reactor thread for 106855us
I20260504 13:56:57.838784 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_0134ffe7086440ccbcadd076d8fe5d01 [id=693e9b6ed89b4ab7b9d5dc7402e17f20]
I20260504 13:56:57.840896 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568]
I20260504 13:56:57.842096 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_f00a5967b1b7422b830dcf11db2acbd8 [id=6a34d9af17c84984b479bbd10025fba5]
I20260504 13:56:57.843325 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_5636294717ff4c63939c2cf9facfc788 [id=6acf451855024161b7826094f8387a9c]
I20260504 13:56:57.861898 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_93e03451ea09440d847c57c026230518 [id=6ad5facbe9ca40eeb7746baef5c566a7]
I20260504 13:56:57.863284 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_8ccb9fc27b374611b20008514d3ad483 [id=6c295b6b72c94ca0b5f1805a555e7f77]
I20260504 13:56:57.873816 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_780e641042db4fd5aeb8af8b1aa08c72 [id=6cb26719d5fe45ccbdef18571c902228]
I20260504 13:56:57.875211 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_6462cb42c34a4eb8976d8d2a4f82d434 [id=6e3151859b6d4da3a477367687862671]
I20260504 13:56:57.876420 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_49715b69b840473b9f7b0fdabcc0084e [id=700b914cd7ec44e791cd0a27d4b9bf81]
I20260504 13:56:57.877712 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_11d8ab61a6044154bb599875a0bc9004 [id=73e12524eb9e416782bf9f7fb58c93bd]
I20260504 13:56:57.885737 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_7dbd11258f34498e82a86f7e3c72f470 [id=7bf52f9bd2a042e59fc4ff612d9846e9]
I20260504 13:56:57.886991 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_df8874a20e834872a12300a4f5628273 [id=7fc0413d79184a2dbe5c470b8506b4e1]
I20260504 13:56:57.888180 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_4842f50414d34bdb8a10802bc059b421 [id=7fc23620ef7342eea52bcce264eb12b3]
I20260504 13:56:57.890098 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_24dab2dd1f45402786d80cba80bbad82 [id=80ca58df22214450b894afea377e9c16]
I20260504 13:56:57.903537 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_cdee25533fe54da0a74b421714b0d7c4 [id=81df524d85774544807ba1902be1aae1]
I20260504 13:56:57.904834 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]
I20260504 13:56:57.914481 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_1c9b2e63142440a5a45018d187a6cded [id=8a427e40b3374dafa140c9156edcc660]
W20260504 13:56:57.920624 9331 leader_election.cc:336] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 806407748d564439807fe5d73db9abda (127.8.238.254:36145): Timed out: RequestConsensusVote RPC to 127.8.238.254:36145 timed out after 0.650s (SENT)
I20260504 13:56:57.921032 9331 leader_election.cc:304] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec9f4c3a43d84fc9a420dd10723c01a1; no voters: 6caedf407d13459e8fd477e8b60b7dc7, 806407748d564439807fe5d73db9abda
I20260504 13:56:57.927989 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_9e9ef3353dcf4b1fadaa2f51bc8f7041 [id=8dcabd7d67334598926f927a62e630d6]
I20260504 13:56:57.929217 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_b59e5e285d50401e94d51afeb4cbc500 [id=8f16af9aec63405cbae2a91b84fa3009]
I20260504 13:56:57.931049 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_a992fb6e038e4408a781d6d8ead37f54 [id=937a02c37dbd45d5bc92979ceaee1c06]
I20260504 13:56:57.940815 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142]
I20260504 13:56:57.941881 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_443402c40ae5443daf4ba4258b847303 [id=98126e5c27cd4d3da57fcd88d481de79]
I20260504 13:56:57.943684 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_10e92469b9cb4e9b965d0a10cfeecca8 [id=99da5017cead4ddeaa3441064b1ed716]
I20260504 13:56:57.956745 11941 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ec9f4c3a43d84fc9a420dd10723c01a1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260504 13:56:57.960816 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_655e2e8cf3544fbeb0ce33829bb2c8a0 [id=9db1e031520d405ea4989e30d5194911]
I20260504 13:56:57.961993 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4]
I20260504 13:56:57.963873 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_225ad80b8318432b88a729064c88e8ec [id=a3cca003025745cebf8e7d403d0778cf]
I20260504 13:56:57.965646 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_e55147137c794be4bff21afce5d36bbc [id=b45733fe3c5547bea00e3967be81ca35]
I20260504 13:56:57.967465 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_6e41cb79b95848ba8c7b1f05f961c7f3 [id=b91fe7c5c3844c109756842f93098699]
I20260504 13:56:57.968544 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13]
I20260504 13:56:57.969663 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_862ae682dfb44737a72e7c5df628a957 [id=bbef146162cb4c30a3ba42ca2a36fe9f]
I20260504 13:56:57.970741 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_5bc32a610edd49ab9641c3726f53f5c6 [id=be5c761518674e7782bfff73366b1675]
I20260504 13:56:58.003938 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_e116f9abbb23419d9a3365f7d0e17db9 [id=bee8acaa102249578514a3d4e5f876d3]
I20260504 13:56:58.005064 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c]
I20260504 13:56:58.032428 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_849bbe7902ca44969c5208828774fa06 [id=c146e84e0e844deda9af40562f980047]
I20260504 13:56:58.035569 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_861ff05bb9e144ea85a3fb7d75f68749 [id=c4b5a9431b8b4d33b41fa27a357e1f46]
I20260504 13:56:58.036599 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_363d6634e42e4dd6a5686c1f0e401ef2 [id=d0d8b54fb69540589420b87f90b4c823]
I20260504 13:56:58.037622 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_3387191fcd6046abbf257447bfbb27f7 [id=d195f5bef6ce49f1b87a4bf5f516eebb]
I20260504 13:56:58.039356 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469]
I20260504 13:56:58.041018 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_6b0124eccb02454b9195b8ddaae5bd38 [id=d47a2c6a9cb34ba4ade10fc23913bf7d]
I20260504 13:56:58.042083 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_d6a5a22d78264e799e80d31d5749600a [id=d63acf266ca440729128a717eed32433]
I20260504 13:56:58.043150 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_91cc6c38c9ff43dca25814664068b1c6 [id=da5c427e75e74ea1ae3a5961ed27e0de]
I20260504 13:56:58.046162 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_8cae213b77994b9fba63b8a7c5f27249 [id=df86bccb4ca8414c90f9fd8753724d89]
I20260504 13:56:58.047293 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1]
I20260504 13:56:58.048506 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_06ecf48a04d94ac8a7e954ff2c260dbf [id=e11be55403d24c8bb35f4708da2e9a78]
I20260504 13:56:58.049772 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_999e37d0dab54ca68c0fa66f74582109 [id=e37fd7e36e604a60832acf303f0ecb7f]
I20260504 13:56:58.050964 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_89c38f9ba46e48fab397a7f3d5d1b805 [id=e4f6621892874f979d1c579d487cba44]
I20260504 13:56:58.052981 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]
I20260504 13:56:58.054248 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_737cc12f07144a6c92dc2974693a649d [id=e9103e8b725b4e81b6cf094395beef3b]
I20260504 13:56:58.056126 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_a5fa0a0bead94aa3b02d6175fefde171 [id=e9fbd43d30ae4d1dbeea534025125b91]
I20260504 13:56:58.057240 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_1e6b4c9e684e45b3b2510d3979c1b5c4 [id=ecfa480ab0ea40008ff4507ed6c5b964]
I20260504 13:56:58.059016 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_2dc0381bd6c84ab59d5bb422de89c0e2 [id=ed756f69850642188b881bd3c680a622]
I20260504 13:56:58.060117 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_09fd612a3c4a4303b8f05530e7d49819 [id=eda14d7d5f7d4391a8c470677ffab4ea]
I20260504 13:56:58.061224 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_72e3a254561b4cae927e6b6bcacd87b6 [id=f227576f105c4b1fbd459b3638ce684b]
I20260504 13:56:58.062292 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_7f912e1d4381428e9572cbb4b13f4eec [id=fb18c55d21704b2ab04dbf35d21377bf]
I20260504 13:56:58.063515 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_9b40ba316993413181a9743fb7bab7fb [id=fb4e5ce5526e44de8effdd766f4c257f]
I20260504 13:56:58.065388 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_7db59fa280b6442b99cf24677b57df83 [id=fdb915ebe83443fd8d2f7f287385f6a4]
I20260504 13:56:58.066474 11781 catalog_manager.cc:679] Loaded metadata for table default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56]
I20260504 13:56:58.090747 11781 tablet_loader.cc:96] loaded metadata for tablet 03e1d998c4314db3b3682a0510393749 (table default.Table_f00a5967b1b7422b830dcf11db2acbd8 [id=6a34d9af17c84984b479bbd10025fba5])
I20260504 13:56:58.109447 11781 tablet_loader.cc:96] loaded metadata for tablet 03fe4ce4f52f48b7b7a804c269e0fd4e (table default.Table_963c93d5d8ed49e6a2052fa1e652da91 [id=399a65bb2fa6428296341de74e964518])
I20260504 13:56:58.112610 11781 tablet_loader.cc:96] loaded metadata for tablet 0be795f04dd54cf19869c937b91d0ddf (table default.Table_64ff08822ec84a56b238186bea0eb82c [id=504a6f156ef14c5f9e68ffc574e0fb0f])
I20260504 13:56:58.114959 11781 tablet_loader.cc:96] loaded metadata for tablet 184fefa651fc411fb5107cc923c02535 (table default.Table_f8a23a8252f84cacb13b338f9d9dc9fc [id=39be2c4d16aa41c0832bfeb23269243a])
I20260504 13:56:58.120530 11781 tablet_loader.cc:96] loaded metadata for tablet 1c12ffa802b840b1937f6f796083c478 (table default.Table_0870a7a3564d42cb8531cae341a9f582 [id=df89117c477d4ef08786e75e2e9a44b1])
I20260504 13:56:58.122205 11781 tablet_loader.cc:96] loaded metadata for tablet 1fbe09b4078c4eebb52d7d00bc87371c (table default.Table_fd3cceda3257414892a1f1e69d4e6344 [id=2560e279f4504b35b3f9309673776eee])
I20260504 13:56:58.123911 11781 tablet_loader.cc:96] loaded metadata for tablet 23ff0d52a90c4056880190dbd936cb8e (table default.Table_24398dd8ee63413ebb585587353ea155 [id=1d2bbdaf6382433cb8c2a55f62f0388d])
I20260504 13:56:58.125461 11781 tablet_loader.cc:96] loaded metadata for tablet 274bb204041c4e1085395c7bd938aaf1 (table default.Table_d0881bfb141241bd8a58a97a29384b23 [id=532ece9a0c134c1f8c1c3793638f34c5])
I20260504 13:56:58.136940 11781 tablet_loader.cc:96] loaded metadata for tablet 2a629c2a69084b6b98984c8d374eeb1e (table default.Table_a5fa0a0bead94aa3b02d6175fefde171 [id=e9fbd43d30ae4d1dbeea534025125b91])
I20260504 13:56:58.138909 11781 tablet_loader.cc:96] loaded metadata for tablet 2c09e9076214405eaa8846fe08158bb3 (table default.Table_6e41cb79b95848ba8c7b1f05f961c7f3 [id=b91fe7c5c3844c109756842f93098699])
I20260504 13:56:58.140117 11781 tablet_loader.cc:96] loaded metadata for tablet 2ddf890c17964e7f90409ebe3c4034ef (table default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e])
I20260504 13:56:58.141294 11781 tablet_loader.cc:96] loaded metadata for tablet 2f6b3e969760442283c3191e062c1839 (table default.Table_5bc32a610edd49ab9641c3726f53f5c6 [id=be5c761518674e7782bfff73366b1675])
I20260504 13:56:58.142515 11781 tablet_loader.cc:96] loaded metadata for tablet 3404ba04d3ec4a44a5e625536b941b58 (table default.Table_7db59fa280b6442b99cf24677b57df83 [id=fdb915ebe83443fd8d2f7f287385f6a4])
I20260504 13:56:58.145107 11781 tablet_loader.cc:96] loaded metadata for tablet 36c8e9ba29a1466082a3e8bedec87b91 (table default.Table_1665799e268a496d9ffc5568fa34a763 [id=403cbbc8a1304e5695a89d3697932c26])
I20260504 13:56:58.146262 11781 tablet_loader.cc:96] loaded metadata for tablet 37b1664312d347288680d97b62c2aecb (table default.Table_861ff05bb9e144ea85a3fb7d75f68749 [id=c4b5a9431b8b4d33b41fa27a357e1f46])
I20260504 13:56:58.149696 11781 tablet_loader.cc:96] loaded metadata for tablet 3d9bd2722d23420ea0eac7664357a60c (table default.Table_821e9f466c974a2fbf24c07f8c266e62 [id=469bd5cb4e7f4a8a9c7af947a834ca7b])
I20260504 13:56:58.154559 11781 tablet_loader.cc:96] loaded metadata for tablet 4a1a77ad56634e068401a2bfbc977761 (table default.Table_9bc232a8e4dd4df1a36b35d09dd50d0a [id=66faab34fb324213a874ea6fae3c96a1])
I20260504 13:56:58.155752 11781 tablet_loader.cc:96] loaded metadata for tablet 4a470ddc5bcb4121bd2f5f02f6e6b945 (table default.Table_13d4530198b743ab8d9c025de85e3ff0 [id=104bbdd0e21b414d917a2a646b912e38])
I20260504 13:56:58.156876 11781 tablet_loader.cc:96] loaded metadata for tablet 4cf1504f0202481ba8b08f9bec5bc3d9 (table default.Table_737cc12f07144a6c92dc2974693a649d [id=e9103e8b725b4e81b6cf094395beef3b])
I20260504 13:56:58.158002 11781 tablet_loader.cc:96] loaded metadata for tablet 4fae9274bb8240798f85b79504a227b3 (table default.Table_e116f9abbb23419d9a3365f7d0e17db9 [id=bee8acaa102249578514a3d4e5f876d3])
I20260504 13:56:58.159096 11781 tablet_loader.cc:96] loaded metadata for tablet 4fdc09a498be470ab04563c62caf7755 (table default.Table_49715b69b840473b9f7b0fdabcc0084e [id=700b914cd7ec44e791cd0a27d4b9bf81])
I20260504 13:56:58.161000 11781 tablet_loader.cc:96] loaded metadata for tablet 5254b1061e2b424487f5d309baaee6b6 (table default.Table_798d2540b8f4489fb11763ccaa36bced [id=049b4fa786fc4dbb8e52931dcaaab90f])
I20260504 13:56:58.162901 11781 tablet_loader.cc:96] loaded metadata for tablet 538c8c2412104c3e8b19394a1cc38818 (table default.Table_7dbd11258f34498e82a86f7e3c72f470 [id=7bf52f9bd2a042e59fc4ff612d9846e9])
I20260504 13:56:58.164819 11781 tablet_loader.cc:96] loaded metadata for tablet 564d0fafb49a4b4bb3bdb240c74c3410 (table default.Table_1c9b2e63142440a5a45018d187a6cded [id=8a427e40b3374dafa140c9156edcc660])
I20260504 13:56:58.165947 11781 tablet_loader.cc:96] loaded metadata for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537])
I20260504 13:56:58.167098 11781 tablet_loader.cc:96] loaded metadata for tablet 59f9115b286d44f0b7eac7c74275cb21 (table default.Table_5944297f8c104978bbc6913ecd8f47a4 [id=3f11131cece44f3ebf3cd0f1e69139ae])
I20260504 13:56:58.168253 11781 tablet_loader.cc:96] loaded metadata for tablet 5cb30835150949f39f6e7e7563381763 (table default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4])
I20260504 13:56:58.170897 11781 tablet_loader.cc:96] loaded metadata for tablet 6323e1c10e7643f8a28f04e285f620e9 (table default.Table_999e37d0dab54ca68c0fa66f74582109 [id=e37fd7e36e604a60832acf303f0ecb7f])
I20260504 13:56:58.172071 11781 tablet_loader.cc:96] loaded metadata for tablet 644170f17f1b45a19677d958b9745e16 (table default.Table_09fd612a3c4a4303b8f05530e7d49819 [id=eda14d7d5f7d4391a8c470677ffab4ea])
I20260504 13:56:58.173965 11781 tablet_loader.cc:96] loaded metadata for tablet 6677a93d01d54f15a506ae1474bcb501 (table default.Table_4842f50414d34bdb8a10802bc059b421 [id=7fc23620ef7342eea52bcce264eb12b3])
I20260504 13:56:58.175904 11781 tablet_loader.cc:96] loaded metadata for tablet 68e92cc3da9b4bcfadb6fa26245f1c99 (table default.Table_18f9a8030cea4cc0abedb81cd5accc59 [id=5c4f226dc33b428e92ca77959e075d7c])
I20260504 13:56:58.177760 11781 tablet_loader.cc:96] loaded metadata for tablet 6dd139c347f7406aac340cc6ac994273 (table default.Table_775e943af0a24713b0aeb8ba35a717c1 [id=ffe434ddd14246c8aa09702c1107af56])
I20260504 13:56:58.179621 11781 tablet_loader.cc:96] loaded metadata for tablet 6f4c8ffb2175493a981c18676485ac04 (table default.Table_93e03451ea09440d847c57c026230518 [id=6ad5facbe9ca40eeb7746baef5c566a7])
I20260504 13:56:58.181916 11781 tablet_loader.cc:96] loaded metadata for tablet 75a43e11c8b34bd7bcddbf6e9b8f30dc (table default.Table_09255abca80f41dc895d81ccd89b9c7a [id=24485cf365ad4f9b89fa0f716fdb34cd])
I20260504 13:56:58.182955 11781 tablet_loader.cc:96] loaded metadata for tablet 7989338bc4bf4bad8fb8fe37587d0a16 (table default.Table_10e92469b9cb4e9b965d0a10cfeecca8 [id=99da5017cead4ddeaa3441064b1ed716])
I20260504 13:56:58.184010 11781 tablet_loader.cc:96] loaded metadata for tablet 7a89bac060214d30924398007dfa1fa1 (table default.Table_2f0fc59ee89c402cb8a0fc23341636e6 [id=00b5922717934e15993bafde9f2513e7])
I20260504 13:56:58.185726 11781 tablet_loader.cc:96] loaded metadata for tablet 7cb6927537704cc4b17617cbdaa21ac1 (table default.Table_d9b090076f5c42d084ebc5b8e8c5bb0c [id=bab1a941deda44fabc9bf67ce6ab0d13])
I20260504 13:56:58.186803 11781 tablet_loader.cc:96] loaded metadata for tablet 7d27b04d3f4443c5ade9616d8626d50e (table default.Table_8ccb9fc27b374611b20008514d3ad483 [id=6c295b6b72c94ca0b5f1805a555e7f77])
I20260504 13:56:58.187845 11781 tablet_loader.cc:96] loaded metadata for tablet 7eaa40eb4b1c425a9edfb7eaba953b65 (table default.Table_f5ef8c8f56c74734b79e268bd64f85e3 [id=2efdf7abcd89420e801bba89304a972e])
I20260504 13:56:58.188889 11781 tablet_loader.cc:96] loaded metadata for tablet 7ed3ee9a13e84594905db47aa33040c2 (table default.Table_6462cb42c34a4eb8976d8d2a4f82d434 [id=6e3151859b6d4da3a477367687862671])
I20260504 13:56:58.190495 11781 tablet_loader.cc:96] loaded metadata for tablet 8702ce3ff1a845f3b70012855cd17efd (table default.Table_24dab2dd1f45402786d80cba80bbad82 [id=80ca58df22214450b894afea377e9c16])
I20260504 13:56:58.191583 11781 tablet_loader.cc:96] loaded metadata for tablet 88619eaa917d4c33a910fc1347ab126e (table default.Table_505ad6df21dd404d8ac2d5b4c150271d [id=c058bce8e15341d0bcbdd6a6b537855c])
I20260504 13:56:58.193358 11781 tablet_loader.cc:96] loaded metadata for tablet 89edd71cf5aa4a6abe2084695d3ed807 (table default.Table_69c9551ae2cf4ef399e40017a33bad73 [id=2d9b8dca85e9477eb77877d5618e9b6f])
I20260504 13:56:58.194511 11781 tablet_loader.cc:96] loaded metadata for tablet 8a985338790e4887af38c49ba013417f (table default.Table_5636294717ff4c63939c2cf9facfc788 [id=6acf451855024161b7826094f8387a9c])
I20260504 13:56:58.196535 11781 tablet_loader.cc:96] loaded metadata for tablet 8c13c4cd2f66426f8e7e81eeac7bc0f3 (table default.Table_c85fc71384a74a73bfe506eab2b8d9b2 [id=9806b0ce05a54e288ce579cde7ec2142])
I20260504 13:56:58.198541 11781 tablet_loader.cc:96] loaded metadata for tablet 8d6db45d2e5c41a3b30cede3301694d3 (table default.Table_862ae682dfb44737a72e7c5df628a957 [id=bbef146162cb4c30a3ba42ca2a36fe9f])
I20260504 13:56:58.199724 11781 tablet_loader.cc:96] loaded metadata for tablet 8db7c4732fa048f9950248f45d144c57 (table default.Table_f8431c2e71824e90844d341601e3be4e [id=5b5423cfb646459cb57b9429e44efc62])
I20260504 13:56:58.201560 11781 tablet_loader.cc:96] loaded metadata for tablet 9077fb9bb1544a2cbeb96f0d11c400db (table default.Table_06ecf48a04d94ac8a7e954ff2c260dbf [id=e11be55403d24c8bb35f4708da2e9a78])
I20260504 13:56:58.203476 11781 tablet_loader.cc:96] loaded metadata for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1])
I20260504 13:56:58.206001 11781 tablet_loader.cc:96] loaded metadata for tablet 975fa1a4975242a388c15fe73abea1c7 (table default.Table_a660204a2583425aa3ecd09c5cc01368 [id=4683a87561f14c049c1ed8eb68dc24d1])
I20260504 13:56:58.207204 11781 tablet_loader.cc:96] loaded metadata for tablet 982f7c53a1f0482e862b53aeeee7cd25 (table default.Table_1e6b4c9e684e45b3b2510d3979c1b5c4 [id=ecfa480ab0ea40008ff4507ed6c5b964])
I20260504 13:56:58.209111 11781 tablet_loader.cc:96] loaded metadata for tablet 99e819494171485e9af8bc8ec33a6ff0 (table default.Table_89c38f9ba46e48fab397a7f3d5d1b805 [id=e4f6621892874f979d1c579d487cba44])
I20260504 13:56:58.210251 11781 tablet_loader.cc:96] loaded metadata for tablet 99fe335b131b45a587ec1cf526dd7f79 (table default.Table_655e2e8cf3544fbeb0ce33829bb2c8a0 [id=9db1e031520d405ea4989e30d5194911])
I20260504 13:56:58.212085 11781 tablet_loader.cc:96] loaded metadata for tablet 9f3905bfb30e4b03a7fcf0a2f57e2117 (table default.Table_597b4ca3e37849d0aa9c86eed64d0614 [id=5a597e1cae1d4126ae355289aef1ced6])
I20260504 13:56:58.213179 11781 tablet_loader.cc:96] loaded metadata for tablet 9f80ce19e8f9498b98952a371476f185 (table default.Table_2bed5ea8f43149e8bc3b21cd9d1b3846 [id=15994ccc978545058e324152f70c8c3e])
I20260504 13:56:58.215061 11781 tablet_loader.cc:96] loaded metadata for tablet a1cf52a9e58744a99dae1958d2c1f867 (table default.Table_6a56efa14aa6497d9926d20508348656 [id=201b906852ca453f9a3f2ae7c8351c81])
I20260504 13:56:58.216251 11781 tablet_loader.cc:96] loaded metadata for tablet a26c6a2ffb8746a3941a56c7259412e3 (table default.Table_225ad80b8318432b88a729064c88e8ec [id=a3cca003025745cebf8e7d403d0778cf])
I20260504 13:56:58.217383 11781 tablet_loader.cc:96] loaded metadata for tablet a2bfb681e6824e9f9bdcbea23061d57a (table default.Table_e55147137c794be4bff21afce5d36bbc [id=b45733fe3c5547bea00e3967be81ca35])
I20260504 13:56:58.220058 11781 tablet_loader.cc:96] loaded metadata for tablet a862a43c5ac443d9bac98e0472ca23b2 (table default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f])
I20260504 13:56:58.221946 11781 tablet_loader.cc:96] loaded metadata for tablet ab4c185332a24d96bc3d0fddc0d20693 (table default.Table_af058f50341e4f478c970f8cb52bf0d7 [id=57ccd7218d454c8c908fe06b798bd83b])
I20260504 13:56:58.223225 11781 tablet_loader.cc:96] loaded metadata for tablet acd5f7d3f7c545d18fe19b64e4bc1de6 (table default.Table_f7b06aadc0024855b08b56cad8c7b6a3 [id=4efd9dd69847446d9f05700d3bd3f914])
I20260504 13:56:58.224429 11781 tablet_loader.cc:96] loaded metadata for tablet af5c5a6cae4e4b6ea3eece2e21a587ec (table default.Table_cdee25533fe54da0a74b421714b0d7c4 [id=81df524d85774544807ba1902be1aae1])
I20260504 13:56:58.227140 11781 tablet_loader.cc:96] loaded metadata for tablet b13303d2a14c4187b82da9f25ce9753e (table default.Table_72e3a254561b4cae927e6b6bcacd87b6 [id=f227576f105c4b1fbd459b3638ce684b])
I20260504 13:56:58.228300 11781 tablet_loader.cc:96] loaded metadata for tablet b320a37b29844c7593a12000eaa3da43 (table default.Table_2099388b6e2a455295be76a6d072758e [id=63e34a66cd644ab7bede27373e321714])
I20260504 13:56:58.230170 11781 tablet_loader.cc:96] loaded metadata for tablet b37a738f2902402ca76e4edebcdba474 (table default.Table_91cc6c38c9ff43dca25814664068b1c6 [id=da5c427e75e74ea1ae3a5961ed27e0de])
I20260504 13:56:58.232307 11781 tablet_loader.cc:96] loaded metadata for tablet b6103d953ace4cb8be3c580b3316f87c (table default.Table_a992fb6e038e4408a781d6d8ead37f54 [id=937a02c37dbd45d5bc92979ceaee1c06])
I20260504 13:56:58.234144 11781 tablet_loader.cc:96] loaded metadata for tablet b7b942037c0e45a6b3cccba288b90b00 (table default.Table_dd11ea89ad7248c39ec114af58cf598f [id=14bd1983ba95468e9e70d8f5724f42e6])
I20260504 13:56:58.235338 11781 tablet_loader.cc:96] loaded metadata for tablet b84c3bedc7a84e0d8a873fb15689b314 (table default.Table_7f912e1d4381428e9572cbb4b13f4eec [id=fb18c55d21704b2ab04dbf35d21377bf])
I20260504 13:56:58.236490 11781 tablet_loader.cc:96] loaded metadata for tablet b8e4239942f34fb081a026a35740e465 (table default.Table_11d8ab61a6044154bb599875a0bc9004 [id=73e12524eb9e416782bf9f7fb58c93bd])
I20260504 13:56:58.237617 11781 tablet_loader.cc:96] loaded metadata for tablet b8ffeae4e5af4cc8a3eb3bc0bedf18cc (table default.Table_003cbc6fdf0b4dddaba008af167dc621 [id=4629003a2d8d461c9e309d49ddecea6a])
I20260504 13:56:58.238744 11781 tablet_loader.cc:96] loaded metadata for tablet ba4584755aae47e0b0f14c6c55474d18 (table default.Table_2d5a582adb634895a55193dd861f3b60 [id=30f349197d564e8da82855fec17b4cd2])
I20260504 13:56:58.239929 11781 tablet_loader.cc:96] loaded metadata for tablet bae90898739b4303af0608e39e8c0367 (table default.Table_073a6dae9da444bea1f96b2f33fe1a85 [id=39c692c399dd43ca963357b6818871cf])
I20260504 13:56:58.241863 11781 tablet_loader.cc:96] loaded metadata for tablet bd635a863f7a41d087947f619e41f95c (table default.Table_849bbe7902ca44969c5208828774fa06 [id=c146e84e0e844deda9af40562f980047])
I20260504 13:56:58.243848 11781 tablet_loader.cc:96] loaded metadata for tablet bdf1c3cef3334a849f7b5e429b7f4990 (table default.Table_42f086bc327c42ccad96925c681a386a [id=2c6aa17153154189a33f5f3aa00b62e9])
I20260504 13:56:58.245003 11781 tablet_loader.cc:96] loaded metadata for tablet c1a01231bb044433ae5827d07a6592f8 (table default.Table_3387191fcd6046abbf257447bfbb27f7 [id=d195f5bef6ce49f1b87a4bf5f516eebb])
I20260504 13:56:58.246124 11781 tablet_loader.cc:96] loaded metadata for tablet c3112c3c3629412e9b76407c7cba8f92 (table default.Table_b59e5e285d50401e94d51afeb4cbc500 [id=8f16af9aec63405cbae2a91b84fa3009])
I20260504 13:56:58.248760 11781 tablet_loader.cc:96] loaded metadata for tablet c47b6e3b4d0c45c28384f7aae6b749de (table default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a])
I20260504 13:56:58.250630 11781 tablet_loader.cc:96] loaded metadata for tablet c6ebc96408d64c2da7dc6c4fc158995c (table default.Table_5ff6dab0e47d41e681533fca02a36153 [id=230a8885dd2b4995815b650d3d9f9f75])
I20260504 13:56:58.251803 11781 tablet_loader.cc:96] loaded metadata for tablet c8989cdde4ca4e349079596e33723b2c (table default.Table_2d928cb3553f40bc8e825d0741ef3199 [id=51b254be98ee457d892039784e4a5f0b])
I20260504 13:56:58.252987 11781 tablet_loader.cc:96] loaded metadata for tablet cc2f95dadd544f57bd6d1467d30e954d (table default.Table_2585508fc4214c8b82d2957644309ac2 [id=256b605e5a2342ed883ec30e3de0ffda])
I20260504 13:56:58.254940 11781 tablet_loader.cc:96] loaded metadata for tablet cf11f17e95dc411294c755e17c57811e (table default.Table_2dc0381bd6c84ab59d5bb422de89c0e2 [id=ed756f69850642188b881bd3c680a622])
I20260504 13:56:58.258405 11781 tablet_loader.cc:96] loaded metadata for tablet dd3c045379734134b49e94340679c11d (table default.Table_443402c40ae5443daf4ba4258b847303 [id=98126e5c27cd4d3da57fcd88d481de79])
I20260504 13:56:58.259635 11781 tablet_loader.cc:96] loaded metadata for tablet ddd30e9259a04d59b5c1bb82aabf7952 (table default.Table_512ae07c2df74cb99bb77006d985f2da [id=09959f81c69c49778a6ee33dc1e85aae])
I20260504 13:56:58.262341 11781 tablet_loader.cc:96] loaded metadata for tablet de6714ee86d64112baf13dd1b2b87582 (table default.Table_50724ddf9a35434385ad5cecc66746d9 [id=d284b13ebceb4a95876bd735a1c65469])
I20260504 13:56:58.263586 11781 tablet_loader.cc:96] loaded metadata for tablet df71dc4f2cf84cda8cfc534a0eea86f8 (table default.Table_1611aa9f4dba40e09dae4433e2d20154 [id=2fb2c56e2fb44b9aac476e06e36e5778])
I20260504 13:56:58.264731 11781 tablet_loader.cc:96] loaded metadata for tablet dffddd4b89984c54bc26dcb4a9488525 (table default.Table_ae10fcacbc05403bb0331bd9c5f8cda4 [id=6a2ae52548034881b7cbb71472132568])
I20260504 13:56:58.265924 11781 tablet_loader.cc:96] loaded metadata for tablet e0b9e47b0a4443c5830422b9d0553446 (table default.Table_4207c9e8019c4088b934376c5568d101 [id=3ca89308de104a5fa27cac4c62047304])
I20260504 13:56:58.267127 11781 tablet_loader.cc:96] loaded metadata for tablet e0e3b36e1f654a3a8df51480801dabd7 (table default.Table_9e9ef3353dcf4b1fadaa2f51bc8f7041 [id=8dcabd7d67334598926f927a62e630d6])
I20260504 13:56:58.268308 11781 tablet_loader.cc:96] loaded metadata for tablet e0e70772b7ce4f27a941918aaf66a865 (table default.Table_60fe6ee7a2414e2aa8ea2d57ee4a827d [id=1de44a1bbf06492084cfd70cf540b213])
I20260504 13:56:58.270053 11781 tablet_loader.cc:96] loaded metadata for tablet e29a7365caf04506b15089f10e6c1677 (table default.Table_6b0124eccb02454b9195b8ddaae5bd38 [id=d47a2c6a9cb34ba4ade10fc23913bf7d])
I20260504 13:56:58.271173 11781 tablet_loader.cc:96] loaded metadata for tablet e53381ee0e7f4e9781b833b435bca831 (table default.Table_9b40ba316993413181a9743fb7bab7fb [id=fb4e5ce5526e44de8effdd766f4c257f])
I20260504 13:56:58.272344 11781 tablet_loader.cc:96] loaded metadata for tablet e5d63f665166466dba1e7e36c5e19506 (table default.Table_363d6634e42e4dd6a5686c1f0e401ef2 [id=d0d8b54fb69540589420b87f90b4c823])
I20260504 13:56:58.273475 11781 tablet_loader.cc:96] loaded metadata for tablet e9e66dacf5794b6195575556be889bce (table default.Table_d6a5a22d78264e799e80d31d5749600a [id=d63acf266ca440729128a717eed32433])
I20260504 13:56:58.276052 11781 tablet_loader.cc:96] loaded metadata for tablet eef975650610423c88ce8e4cfb0b013b (table default.Table_0134ffe7086440ccbcadd076d8fe5d01 [id=693e9b6ed89b4ab7b9d5dc7402e17f20])
I20260504 13:56:58.277849 11781 tablet_loader.cc:96] loaded metadata for tablet f337e9698fe64282a684e90d9c61da5a (table default.Table_35722081733246ae8ca41a5303921ffe [id=3ff4d3cfe6754e75bf62225a5a2f2d12])
I20260504 13:56:58.279006 11781 tablet_loader.cc:96] loaded metadata for tablet f3e86027e3e74bc196e9a9ffc9fdf8a5 (table default.Table_780e641042db4fd5aeb8af8b1aa08c72 [id=6cb26719d5fe45ccbdef18571c902228])
I20260504 13:56:58.280253 11781 tablet_loader.cc:96] loaded metadata for tablet f71f444ce9214e3f98eff6ac8ca7dde7 (table default.Table_df8874a20e834872a12300a4f5628273 [id=7fc0413d79184a2dbe5c470b8506b4e1])
I20260504 13:56:58.281400 11781 tablet_loader.cc:96] loaded metadata for tablet fac3e36cfa4848a1ab015f89c56c2740 (table default.Table_8cae213b77994b9fba63b8a7c5f27249 [id=df86bccb4ca8414c90f9fd8753724d89])
I20260504 13:56:58.282569 11781 tablet_loader.cc:96] loaded metadata for tablet fe7ab3b577b74a9c9a40cb15c94f8aa1 (table default.Table_dee1335eff89438e9c68835a4c258031 [id=1714df2b31ff4865b6b357500d121a26])
W20260504 13:56:58.304160 11781 catalog_manager.cc:1486] Time spent T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: Loading table and tablet metadata into memory: real 6.234s user 0.365s sys 0.093s
I20260504 13:56:58.304513 11781 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260504 13:56:58.304294 9195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
W20260504 13:56:58.305471 9195 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35460 (request call id 545) took 4036 ms (client timeout 654 ms). Trace:
W20260504 13:56:58.305840 9195 rpcz_store.cc:269] 0504 13:56:54.269256 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:54.269900 (+ 644us) service_pool.cc:224] Handling call
0504 13:56:58.305373 (+4035473us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:58.306897 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
W20260504 13:56:58.307976 9198 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58048 (request call id 10) took 5808 ms (client timeout 721 ms). Trace:
W20260504 13:56:58.308264 9198 rpcz_store.cc:269] 0504 13:56:52.499668 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:52.499790 (+ 122us) service_pool.cc:224] Handling call
0504 13:56:58.307595 (+5807805us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 13:56:58.309088 9198 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35460 (request call id 548) took 1039 ms (client timeout 650 ms). Trace:
W20260504 13:56:58.309377 9198 rpcz_store.cc:269] 0504 13:56:57.269404 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:58.308882 (+1039478us) service_pool.cc:211] Skipping call since client already timed out
0504 13:56:58.309022 (+ 140us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260504 13:56:58.310073 9198 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
I20260504 13:56:58.310997 9199 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
W20260504 13:56:58.311694 9199 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35460 (request call id 543) took 5768 ms (client timeout 516 ms). Trace:
W20260504 13:56:58.311993 9199 rpcz_store.cc:269] 0504 13:56:52.543746 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:52.543947 (+ 201us) service_pool.cc:224] Handling call
0504 13:56:58.311636 (+5767689us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:58.312773 9197 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
W20260504 13:56:58.313433 9197 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35460 (request call id 544) took 5224 ms (client timeout 516 ms). Trace:
W20260504 13:56:58.313714 9197 rpcz_store.cc:269] 0504 13:56:53.089165 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:53.089360 (+ 195us) service_pool.cc:224] Handling call
0504 13:56:58.313369 (+5224009us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:58.314503 9196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
W20260504 13:56:58.315308 9196 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58048 (request call id 11) took 5053 ms (client timeout 721 ms). Trace:
W20260504 13:56:58.315627 9196 rpcz_store.cc:269] 0504 13:56:53.261733 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:53.261932 (+ 199us) service_pool.cc:224] Handling call
0504 13:56:58.315245 (+5053313us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:58.316416 9190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
W20260504 13:56:58.317143 9190 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58048 (request call id 14) took 1438 ms (client timeout 730 ms). Trace:
W20260504 13:56:58.317487 9190 rpcz_store.cc:269] 0504 13:56:56.878428 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:56.879014 (+ 586us) service_pool.cc:224] Handling call
0504 13:56:58.317076 (+1438062us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:58.318684 9191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
W20260504 13:56:58.319489 9191 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35460 (request call id 547) took 1719 ms (client timeout 650 ms). Trace:
W20260504 13:56:58.319810 9191 rpcz_store.cc:269] 0504 13:56:56.600562 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:56.602254 (+ 1692us) service_pool.cc:224] Handling call
0504 13:56:58.319434 (+1717180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:58.321239 9192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
W20260504 13:56:58.322006 9192 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58048 (request call id 13) took 3100 ms (client timeout 594 ms). Trace:
W20260504 13:56:58.322315 9192 rpcz_store.cc:269] 0504 13:56:55.221587 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:55.221838 (+ 251us) service_pool.cc:224] Handling call
0504 13:56:58.321951 (+3100113us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:58.323119 9193 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec9f4c3a43d84fc9a420dd10723c01a1" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
W20260504 13:56:58.323890 9193 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35460 (request call id 546) took 3378 ms (client timeout 654 ms). Trace:
W20260504 13:56:58.324215 9193 rpcz_store.cc:269] 0504 13:56:54.945182 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:54.945362 (+ 180us) service_pool.cc:224] Handling call
0504 13:56:58.323834 (+3378472us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:58.325425 9182 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:45802 (request call id 44) took 6252 ms. Trace:
I20260504 13:56:58.325709 9182 rpcz_store.cc:276] 0504 13:56:52.073341 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:52.075284 (+ 1943us) service_pool.cc:224] Handling call
0504 13:56:58.325235 (+6249951us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:58.320572 9194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6caedf407d13459e8fd477e8b60b7dc7" candidate_term: 7 candidate_status { last_received { term: 6 index: 378 } } ignore_live_leader: false dest_uuid: "806407748d564439807fe5d73db9abda" is_pre_election: true
W20260504 13:56:58.327025 9194 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58048 (request call id 12) took 3765 ms (client timeout 594 ms). Trace:
W20260504 13:56:58.327420 9194 rpcz_store.cc:269] 0504 13:56:54.561685 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:54.561866 (+ 181us) service_pool.cc:224] Handling call
0504 13:56:58.326967 (+3765101us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260504 13:56:58.310679 9198 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58048 (request call id 15) took 635 ms (client timeout 730 ms). Trace:
W20260504 13:56:58.331183 9198 rpcz_store.cc:269] 0504 13:56:57.675778 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:56:58.309900 (+634122us) service_pool.cc:224] Handling call
0504 13:56:58.310630 (+ 730us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:56:58.332204 10690 leader_election.cc:304] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6caedf407d13459e8fd477e8b60b7dc7; no voters: 806407748d564439807fe5d73db9abda, ec9f4c3a43d84fc9a420dd10723c01a1
I20260504 13:56:58.371304 11781 catalog_manager.cc:1269] Loaded cluster ID: 302d3e241578494980ac3509cf5306fb
I20260504 13:56:58.371601 11781 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260504 13:56:58.408615 11781 catalog_manager.cc:1514] Loading token signing keys...
I20260504 13:56:58.414718 11942 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6caedf407d13459e8fd477e8b60b7dc7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260504 13:56:58.415663 10690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.238.254:36145, user_credentials={real_user=slave}} blocked reactor thread for 83542.7us
I20260504 13:56:58.428328 11781 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: Loaded TSK: 0
I20260504 13:56:58.429728 11781 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260504 13:56:58.510370 9182 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45802:
name: "default.Table_22baa507648b4359a493e748b9fe77d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:58.512398 9189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45802:
name: "default.Table_3aa87975e1fe4f5f89021c21788fb1ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:56:58.513445 9186 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45802:
table { table_name: "default.Table_24398dd8ee63413ebb585587353ea155" } new_table_name: "default.Table_69c53f858e044915b4cc201c8450f4b6" modify_external_catalogs: true
W20260504 13:56:58.514011 9189 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3aa87975e1fe4f5f89021c21788fb1ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:56:58.514462 9182 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22baa507648b4359a493e748b9fe77d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:58.515008 9181 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45802:
table { table_name: "default.Table_2d5a582adb634895a55193dd861f3b60" } modify_external_catalogs: true
I20260504 13:56:58.515712 9181 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45802:
table { table_name: "default.Table_2d5a582adb634895a55193dd861f3b60" } modify_external_catalogs: true
I20260504 13:56:58.521138 9188 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45802:
name: "default.Table_b6636be462a045f390318cfe598f8f1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:58.522750 9188 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6636be462a045f390318cfe598f8f1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:58.528606 9180 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45802:
name: "default.Table_32b65abc85d34dd580150ae093d61be7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:56:58.530138 9180 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32b65abc85d34dd580150ae093d61be7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:56:58.541396 9185 master_service.cc:954] ReplaceTablet: received request to replace tablet 68e92cc3da9b4bcfadb6fa26245f1c99 from {username='slave'} at 127.0.0.1:45802
I20260504 13:56:58.543206 9183 master_service.cc:954] ReplaceTablet: received request to replace tablet c6ebc96408d64c2da7dc6c4fc158995c from {username='slave'} at 127.0.0.1:45802
W20260504 13:56:58.662264 9186 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23ff0d52a90c4056880190dbd936cb8e consensus state has no leader
W20260504 13:56:58.705441 9176 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 23ff0d52a90c4056880190dbd936cb8e (table default.Table_69c53f858e044915b4cc201c8450f4b6 [id=1d2bbdaf6382433cb8c2a55f62f0388d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 23ff0d52a90c4056880190dbd936cb8e consensus state has no leader
I20260504 13:56:58.748095 9183 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: Sending DeleteTablet for 3 replicas of tablet c6ebc96408d64c2da7dc6c4fc158995c
I20260504 13:56:58.749584 9183 catalog_manager.cc:6741] ReplaceTablet: tablet c6ebc96408d64c2da7dc6c4fc158995c deleted and replaced by tablet c69457ad1b93443c91f33a82ca91199b
I20260504 13:56:58.749756 9805 tablet_service.cc:1558] Processing DeleteTablet for tablet c6ebc96408d64c2da7dc6c4fc158995c with delete_type TABLET_DATA_DELETED (replaced by tablet c69457ad1b93443c91f33a82ca91199b) from {username='slave'} at 127.0.0.1:40598
I20260504 13:56:58.752032 9507 tablet_service.cc:1558] Processing DeleteTablet for tablet c6ebc96408d64c2da7dc6c4fc158995c with delete_type TABLET_DATA_DELETED (replaced by tablet c69457ad1b93443c91f33a82ca91199b) from {username='slave'} at 127.0.0.1:40114
I20260504 13:56:58.755599 9659 tablet_service.cc:1558] Processing DeleteTablet for tablet c6ebc96408d64c2da7dc6c4fc158995c with delete_type TABLET_DATA_DELETED (replaced by tablet c69457ad1b93443c91f33a82ca91199b) from {username='slave'} at 127.0.0.1:34768
I20260504 13:56:58.758718 9147 master-stress-test.cc:579] Killed test thread 1/10
I20260504 13:56:58.759068 9147 master-stress-test.cc:577] Killing test thread 2/10
I20260504 13:56:58.761909 11961 tablet_replica.cc:333] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:58.765172 11960 tablet_replica.cc:333] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:58.777194 11961 raft_consensus.cc:2243] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:58.782512 11961 raft_consensus.cc:2272] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:58.790474 11961 ts_tablet_manager.cc:1916] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:58.805631 11960 raft_consensus.cc:2243] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:58.806232 11960 raft_consensus.cc:2272] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:58.808521 11960 ts_tablet_manager.cc:1916] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:58.868381 11960 ts_tablet_manager.cc:1929] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:58.870023 11960 log.cc:1199] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/c6ebc96408d64c2da7dc6c4fc158995c
I20260504 13:56:58.871225 11960 ts_tablet_manager.cc:1950] T c6ebc96408d64c2da7dc6c4fc158995c P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:58.874079 11961 ts_tablet_manager.cc:1929] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:58.877149 11961 log.cc:1199] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/c6ebc96408d64c2da7dc6c4fc158995c
I20260504 13:56:58.878906 11961 ts_tablet_manager.cc:1950] T c6ebc96408d64c2da7dc6c4fc158995c P 7b79fe7dd8934dd996393a2d96eabf00: Deleting consensus metadata
I20260504 13:56:58.880617 9175 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet c6ebc96408d64c2da7dc6c4fc158995c (table default.Table_5ff6dab0e47d41e681533fca02a36153 [id=230a8885dd2b4995815b650d3d9f9f75]) successfully deleted
W20260504 13:56:58.981315 9173 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.238.253:36337, user_credentials={real_user=slave}} blocked reactor thread for 216734us
I20260504 13:56:58.984279 9147 master-stress-test.cc:579] Killed test thread 2/10
I20260504 13:56:58.984534 9147 master-stress-test.cc:577] Killing test thread 3/10
I20260504 13:56:58.984743 9147 master-stress-test.cc:579] Killed test thread 3/10
I20260504 13:56:58.984943 9147 master-stress-test.cc:577] Killing test thread 4/10
I20260504 13:56:58.991012 9173 catalog_manager.cc:5002] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): tablet c6ebc96408d64c2da7dc6c4fc158995c (table default.Table_5ff6dab0e47d41e681533fca02a36153 [id=230a8885dd2b4995815b650d3d9f9f75]) successfully deleted
I20260504 13:56:59.010926 11962 tablet_replica.cc:333] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:59.011969 11962 raft_consensus.cc:2243] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:59.012730 11962 raft_consensus.cc:2272] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:59.020000 11962 ts_tablet_manager.cc:1916] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:59.057775 9147 master-stress-test.cc:579] Killed test thread 4/10
I20260504 13:56:59.058058 9147 master-stress-test.cc:577] Killing test thread 5/10
I20260504 13:56:59.058288 9147 master-stress-test.cc:579] Killed test thread 5/10
I20260504 13:56:59.058493 9147 master-stress-test.cc:577] Killing test thread 6/10
I20260504 13:56:59.058972 9147 master-stress-test.cc:579] Killed test thread 6/10
I20260504 13:56:59.059204 9147 master-stress-test.cc:577] Killing test thread 7/10
I20260504 13:56:59.061573 9181 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: Sending DeleteTablet for 3 replicas of tablet ba4584755aae47e0b0f14c6c55474d18
I20260504 13:56:59.064934 9659 tablet_service.cc:1558] Processing DeleteTablet for tablet ba4584755aae47e0b0f14c6c55474d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:58 UTC) from {username='slave'} at 127.0.0.1:34768
I20260504 13:56:59.065523 9507 tablet_service.cc:1558] Processing DeleteTablet for tablet ba4584755aae47e0b0f14c6c55474d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:58 UTC) from {username='slave'} at 127.0.0.1:40114
I20260504 13:56:59.066191 11961 tablet_replica.cc:333] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00: stopping tablet replica
I20260504 13:56:59.067095 11961 raft_consensus.cc:2243] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:56:59.067277 9185 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: Sending DeleteTablet for 3 replicas of tablet 68e92cc3da9b4bcfadb6fa26245f1c99
I20260504 13:56:59.067916 11961 raft_consensus.cc:2272] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:59.068599 9185 catalog_manager.cc:6741] ReplaceTablet: tablet 68e92cc3da9b4bcfadb6fa26245f1c99 deleted and replaced by tablet 799a311f768f4b1282a17ac5b64f97b7
I20260504 13:56:59.069991 11961 ts_tablet_manager.cc:1916] T ba4584755aae47e0b0f14c6c55474d18 P 7b79fe7dd8934dd996393a2d96eabf00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:59.070955 9659 tablet_service.cc:1558] Processing DeleteTablet for tablet 68e92cc3da9b4bcfadb6fa26245f1c99 with delete_type TABLET_DATA_DELETED (replaced by tablet 799a311f768f4b1282a17ac5b64f97b7) from {username='slave'} at 127.0.0.1:34768
I20260504 13:56:59.072798 9507 tablet_service.cc:1558] Processing DeleteTablet for tablet 68e92cc3da9b4bcfadb6fa26245f1c99 with delete_type TABLET_DATA_DELETED (replaced by tablet 799a311f768f4b1282a17ac5b64f97b7) from {username='slave'} at 127.0.0.1:40114
I20260504 13:56:59.089931 9147 master-stress-test.cc:579] Killed test thread 7/10
I20260504 13:56:59.090234 9147 master-stress-test.cc:577] Killing test thread 8/10
I20260504 13:56:59.090552 9147 master-stress-test.cc:579] Killed test thread 8/10
I20260504 13:56:59.090772 9147 master-stress-test.cc:577] Killing test thread 9/10
I20260504 13:56:59.091070 9147 master-stress-test.cc:579] Killed test thread 9/10
I20260504 13:56:59.091365 9147 master-stress-test.cc:577] Killing test thread 10/10
I20260504 13:56:59.091740 9147 master-stress-test.cc:579] Killed test thread 10/10
I20260504 13:56:59.092142 9147 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu with pid 9445
I20260504 13:56:59.119576 9659 tablet_service.cc:1511] Processing CreateTablet for tablet a5f2bfe937b34c33bad8b736384248f9 (DEFAULT_TABLE table=default.Table_b6636be462a045f390318cfe598f8f1e [id=a38e43964aae4653887dbef721165253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:59.120850 9659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5f2bfe937b34c33bad8b736384248f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:59.122774 9657 tablet_service.cc:1511] Processing CreateTablet for tablet ba270b1227d14c60bd07046a73c6b7ec (DEFAULT_TABLE table=default.Table_32b65abc85d34dd580150ae093d61be7 [id=f28bb1823d83446a8df950c9c1394c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:59.123948 9657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba270b1227d14c60bd07046a73c6b7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:59.124619 9660 tablet_service.cc:1511] Processing CreateTablet for tablet 0bb1a301d1504d89973e8b17c39c2e8a (DEFAULT_TABLE table=default.Table_22baa507648b4359a493e748b9fe77d8 [id=147ea4c145504ddca2861099be3ebc43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:59.125577 9660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bb1a301d1504d89973e8b17c39c2e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:59.127550 9666 tablet_service.cc:1511] Processing CreateTablet for tablet c69457ad1b93443c91f33a82ca91199b (DEFAULT_TABLE table=default.Table_5ff6dab0e47d41e681533fca02a36153 [id=230a8885dd2b4995815b650d3d9f9f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:59.128831 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69457ad1b93443c91f33a82ca91199b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:59.145817 11967 tablet_bootstrap.cc:492] T ba270b1227d14c60bd07046a73c6b7ec P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:59.153415 11967 tablet_bootstrap.cc:654] T ba270b1227d14c60bd07046a73c6b7ec P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:59.173029 11962 ts_tablet_manager.cc:1929] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:59.173496 11962 log.cc:1199] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/c6ebc96408d64c2da7dc6c4fc158995c
I20260504 13:56:59.174642 11962 ts_tablet_manager.cc:1950] T c6ebc96408d64c2da7dc6c4fc158995c P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:59.200497 11962 tablet_replica.cc:333] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:59.201896 11962 raft_consensus.cc:2243] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:59.202849 11962 raft_consensus.cc:2272] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:59.204116 9174 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet c6ebc96408d64c2da7dc6c4fc158995c (table default.Table_5ff6dab0e47d41e681533fca02a36153 [id=230a8885dd2b4995815b650d3d9f9f75]) successfully deleted
I20260504 13:56:59.204608 9666 tablet_service.cc:1511] Processing CreateTablet for tablet 70db1f7c49e54097a5917d679071f505 (DEFAULT_TABLE table=default.Table_3aa87975e1fe4f5f89021c21788fb1ff [id=0291efa6363d4c208c98c80dbcff15bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:59.204964 9659 tablet_service.cc:1511] Processing CreateTablet for tablet 799a311f768f4b1282a17ac5b64f97b7 (DEFAULT_TABLE table=default.Table_18f9a8030cea4cc0abedb81cd5accc59 [id=5c4f226dc33b428e92ca77959e075d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:59.205587 9666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70db1f7c49e54097a5917d679071f505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:59.206959 9659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799a311f768f4b1282a17ac5b64f97b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:59.216804 11962 ts_tablet_manager.cc:1916] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:59.222609 9736 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:59.232689 11967 tablet_bootstrap.cc:492] T ba270b1227d14c60bd07046a73c6b7ec P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:59.233192 11967 ts_tablet_manager.cc:1403] T ba270b1227d14c60bd07046a73c6b7ec P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.088s user 0.019s sys 0.012s
I20260504 13:56:59.245966 11967 raft_consensus.cc:359] T ba270b1227d14c60bd07046a73c6b7ec P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.246893 11967 raft_consensus.cc:385] T ba270b1227d14c60bd07046a73c6b7ec P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:59.247264 11967 raft_consensus.cc:740] T ba270b1227d14c60bd07046a73c6b7ec P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:59.248176 11967 consensus_queue.cc:260] T ba270b1227d14c60bd07046a73c6b7ec P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.250365 11967 ts_tablet_manager.cc:1434] T ba270b1227d14c60bd07046a73c6b7ec P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260504 13:56:59.252218 11967 tablet_bootstrap.cc:492] T a5f2bfe937b34c33bad8b736384248f9 P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
I20260504 13:56:59.255326 9805 tablet_service.cc:1558] Processing DeleteTablet for tablet ba4584755aae47e0b0f14c6c55474d18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-04 13:56:58 UTC) from {username='slave'} at 127.0.0.1:40598
I20260504 13:56:59.256402 11960 tablet_replica.cc:333] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:59.257520 9805 tablet_service.cc:1558] Processing DeleteTablet for tablet 68e92cc3da9b4bcfadb6fa26245f1c99 with delete_type TABLET_DATA_DELETED (replaced by tablet 799a311f768f4b1282a17ac5b64f97b7) from {username='slave'} at 127.0.0.1:40598
I20260504 13:56:59.258193 11967 tablet_bootstrap.cc:654] T a5f2bfe937b34c33bad8b736384248f9 P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:59.259354 9805 tablet_service.cc:1511] Processing CreateTablet for tablet a5f2bfe937b34c33bad8b736384248f9 (DEFAULT_TABLE table=default.Table_b6636be462a045f390318cfe598f8f1e [id=a38e43964aae4653887dbef721165253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:59.260701 9805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5f2bfe937b34c33bad8b736384248f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:59.270898 9801 tablet_service.cc:1511] Processing CreateTablet for tablet ba270b1227d14c60bd07046a73c6b7ec (DEFAULT_TABLE table=default.Table_32b65abc85d34dd580150ae093d61be7 [id=f28bb1823d83446a8df950c9c1394c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:59.271891 11967 tablet_bootstrap.cc:492] T a5f2bfe937b34c33bad8b736384248f9 P 5beceb6317444247b31b44a6581707c6: No bootstrap required, opened a new log
I20260504 13:56:59.271996 9801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba270b1227d14c60bd07046a73c6b7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:59.272245 11967 ts_tablet_manager.cc:1403] T a5f2bfe937b34c33bad8b736384248f9 P 5beceb6317444247b31b44a6581707c6: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.007s
I20260504 13:56:59.274657 11967 raft_consensus.cc:359] T a5f2bfe937b34c33bad8b736384248f9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.275251 11967 raft_consensus.cc:385] T a5f2bfe937b34c33bad8b736384248f9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:59.275521 11967 raft_consensus.cc:740] T a5f2bfe937b34c33bad8b736384248f9 P 5beceb6317444247b31b44a6581707c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beceb6317444247b31b44a6581707c6, State: Initialized, Role: FOLLOWER
I20260504 13:56:59.276086 11967 consensus_queue.cc:260] T a5f2bfe937b34c33bad8b736384248f9 P 5beceb6317444247b31b44a6581707c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.278625 11967 ts_tablet_manager.cc:1434] T a5f2bfe937b34c33bad8b736384248f9 P 5beceb6317444247b31b44a6581707c6: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260504 13:56:59.284515 9797 tablet_service.cc:1511] Processing CreateTablet for tablet 0bb1a301d1504d89973e8b17c39c2e8a (DEFAULT_TABLE table=default.Table_22baa507648b4359a493e748b9fe77d8 [id=147ea4c145504ddca2861099be3ebc43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:59.285660 9797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bb1a301d1504d89973e8b17c39c2e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:59.291531 11962 ts_tablet_manager.cc:1929] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:59.291962 11962 log.cc:1199] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/ba4584755aae47e0b0f14c6c55474d18
I20260504 13:56:59.292701 11962 ts_tablet_manager.cc:1950] T ba4584755aae47e0b0f14c6c55474d18 P 5beceb6317444247b31b44a6581707c6: Deleting consensus metadata
I20260504 13:56:59.306188 9806 tablet_service.cc:1511] Processing CreateTablet for tablet c69457ad1b93443c91f33a82ca91199b (DEFAULT_TABLE table=default.Table_5ff6dab0e47d41e681533fca02a36153 [id=230a8885dd2b4995815b650d3d9f9f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:59.308398 9795 tablet_service.cc:1511] Processing CreateTablet for tablet 70db1f7c49e54097a5917d679071f505 (DEFAULT_TABLE table=default.Table_3aa87975e1fe4f5f89021c21788fb1ff [id=0291efa6363d4c208c98c80dbcff15bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:59.311040 9799 tablet_service.cc:1511] Processing CreateTablet for tablet 799a311f768f4b1282a17ac5b64f97b7 (DEFAULT_TABLE table=default.Table_18f9a8030cea4cc0abedb81cd5accc59 [id=5c4f226dc33b428e92ca77959e075d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:56:59.319137 9806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69457ad1b93443c91f33a82ca91199b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:59.320654 9795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70db1f7c49e54097a5917d679071f505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:59.322142 9799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799a311f768f4b1282a17ac5b64f97b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:56:59.378676 11960 raft_consensus.cc:2243] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:59.379427 11960 raft_consensus.cc:2272] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:59.382306 11960 ts_tablet_manager.cc:1916] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:56:59.409081 11967 tablet_bootstrap.cc:492] T c69457ad1b93443c91f33a82ca91199b P 5beceb6317444247b31b44a6581707c6: Bootstrap starting.
W20260504 13:56:59.424649 9173 connection.cc:570] client connection to 127.8.238.193:33117 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:56:59.425032 9173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
I20260504 13:56:59.431399 11967 tablet_bootstrap.cc:654] T c69457ad1b93443c91f33a82ca91199b P 5beceb6317444247b31b44a6581707c6: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:59.432054 9618 connection.cc:570] client connection to 127.8.238.193:33117 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:56:59.432548 9618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260504 13:56:59.437343 11962 tablet_replica.cc:333] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6: stopping tablet replica
I20260504 13:56:59.438517 11962 raft_consensus.cc:2243] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:59.439931 9174 catalog_manager.cc:5002] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): tablet ba4584755aae47e0b0f14c6c55474d18 (table default.Table_2d5a582adb634895a55193dd861f3b60 [id=30f349197d564e8da82855fec17b4cd2]) successfully deleted
I20260504 13:56:59.439024 11962 raft_consensus.cc:2272] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:59.443650 11975 tablet_bootstrap.cc:492] T 70db1f7c49e54097a5917d679071f505 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:59.444190 11962 ts_tablet_manager.cc:1916] T 68e92cc3da9b4bcfadb6fa26245f1c99 P 5beceb6317444247b31b44a6581707c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260504 13:56:59.448112 9620 connection.cc:570] server connection from 127.8.238.193:37485 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:56:59.447606 9770 connection.cc:570] server connection from 127.8.238.193:54761 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:56:59.453043 9767 connection.cc:570] client connection to 127.8.238.193:33117 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:56:59.453467 9767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260504 13:56:59.459801 11960 ts_tablet_manager.cc:1929] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:59.460240 11960 log.cc:1199] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/ba4584755aae47e0b0f14c6c55474d18
I20260504 13:56:59.463939 11975 tablet_bootstrap.cc:654] T 70db1f7c49e54097a5917d679071f505 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:59.470602 11960 ts_tablet_manager.cc:1950] T ba4584755aae47e0b0f14c6c55474d18 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:59.473241 9147 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu with pid 9586
I20260504 13:56:59.474960 11960 tablet_replica.cc:333] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888: stopping tablet replica
I20260504 13:56:59.475072 9175 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet ba4584755aae47e0b0f14c6c55474d18 (table default.Table_2d5a582adb634895a55193dd861f3b60 [id=30f349197d564e8da82855fec17b4cd2]) successfully deleted
I20260504 13:56:59.476027 11960 raft_consensus.cc:2243] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:56:59.476646 11960 raft_consensus.cc:2272] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:56:59.478892 11960 ts_tablet_manager.cc:1916] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260504 13:56:59.482161 9173 catalog_manager.cc:4729] TS 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba4584755aae47e0b0f14c6c55474d18: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111)
I20260504 13:56:59.515187 11975 tablet_bootstrap.cc:492] T 70db1f7c49e54097a5917d679071f505 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:59.516762 11975 ts_tablet_manager.cc:1403] T 70db1f7c49e54097a5917d679071f505 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.073s user 0.023s sys 0.005s
W20260504 13:56:59.521108 9767 consensus_peers.cc:597] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.521624 9767 consensus_peers.cc:597] T af5c5a6cae4e4b6ea3eece2e21a587ec P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.521999 9767 consensus_peers.cc:597] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.522367 9767 consensus_peers.cc:597] T 8db7c4732fa048f9950248f45d144c57 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.522725 9767 consensus_peers.cc:597] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.523115 9767 consensus_peers.cc:597] T e0b9e47b0a4443c5830422b9d0553446 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.523542 9767 consensus_peers.cc:597] T 2ddf890c17964e7f90409ebe3c4034ef P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.523903 9767 consensus_peers.cc:597] T c47b6e3b4d0c45c28384f7aae6b749de P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.524246 9767 consensus_peers.cc:597] T c8989cdde4ca4e349079596e33723b2c P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.524588 9767 consensus_peers.cc:597] T e9e66dacf5794b6195575556be889bce P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.524897 9767 consensus_peers.cc:597] T 0be795f04dd54cf19869c937b91d0ddf P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.525225 9767 consensus_peers.cc:597] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.525553 9767 consensus_peers.cc:597] T 4a1a77ad56634e068401a2bfbc977761 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:59.528613 11960 ts_tablet_manager.cc:1929] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:56:59.529670 11960 log.cc:1199] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/68e92cc3da9b4bcfadb6fa26245f1c99
I20260504 13:56:59.530485 11960 ts_tablet_manager.cc:1950] T 68e92cc3da9b4bcfadb6fa26245f1c99 P d4051d2bf0bf410eb8acd107560df888: Deleting consensus metadata
I20260504 13:56:59.526831 11975 raft_consensus.cc:359] T 70db1f7c49e54097a5917d679071f505 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.533186 11975 raft_consensus.cc:385] T 70db1f7c49e54097a5917d679071f505 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:59.533959 9175 catalog_manager.cc:5002] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563): tablet 68e92cc3da9b4bcfadb6fa26245f1c99 (table default.Table_18f9a8030cea4cc0abedb81cd5accc59 [id=5c4f226dc33b428e92ca77959e075d7c]) successfully deleted
I20260504 13:56:59.533567 11975 raft_consensus.cc:740] T 70db1f7c49e54097a5917d679071f505 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:59.535012 11975 consensus_queue.cc:260] T 70db1f7c49e54097a5917d679071f505 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.536566 9881 heartbeater.cc:499] Master 127.8.238.254:36145 was elected leader, sending a full tablet report...
I20260504 13:56:59.549636 11975 ts_tablet_manager.cc:1434] T 70db1f7c49e54097a5917d679071f505 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.032s user 0.001s sys 0.005s
W20260504 13:56:59.553553 9767 consensus_peers.cc:597] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.554091 9767 consensus_peers.cc:597] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:59.558008 11975 tablet_bootstrap.cc:492] T 799a311f768f4b1282a17ac5b64f97b7 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:59.565021 11975 tablet_bootstrap.cc:654] T 799a311f768f4b1282a17ac5b64f97b7 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:59.582782 9767 consensus_peers.cc:597] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:59.584443 11975 tablet_bootstrap.cc:492] T 799a311f768f4b1282a17ac5b64f97b7 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:59.584935 11975 ts_tablet_manager.cc:1403] T 799a311f768f4b1282a17ac5b64f97b7 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.009s
I20260504 13:56:59.587580 11975 raft_consensus.cc:359] T 799a311f768f4b1282a17ac5b64f97b7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.591054 11975 raft_consensus.cc:385] T 799a311f768f4b1282a17ac5b64f97b7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:59.591411 11975 raft_consensus.cc:740] T 799a311f768f4b1282a17ac5b64f97b7 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:59.592160 11975 consensus_queue.cc:260] T 799a311f768f4b1282a17ac5b64f97b7 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.594640 11975 ts_tablet_manager.cc:1434] T 799a311f768f4b1282a17ac5b64f97b7 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
W20260504 13:56:59.605728 9767 consensus_peers.cc:597] T 23ff0d52a90c4056880190dbd936cb8e P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:59.611372 11975 tablet_bootstrap.cc:492] T c69457ad1b93443c91f33a82ca91199b P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:59.615684 9767 consensus_peers.cc:597] T f71f444ce9214e3f98eff6ac8ca7dde7 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.616112 9767 consensus_peers.cc:597] T 5254b1061e2b424487f5d309baaee6b6 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:59.629022 11975 tablet_bootstrap.cc:654] T c69457ad1b93443c91f33a82ca91199b P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:59.635380 9767 consensus_peers.cc:597] T 99fe335b131b45a587ec1cf526dd7f79 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.646355 9767 consensus_peers.cc:597] T 6f4c8ffb2175493a981c18676485ac04 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.655800 9767 consensus_peers.cc:597] T 3d9bd2722d23420ea0eac7664357a60c P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.664945 9767 consensus_peers.cc:597] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.665320 9767 consensus_peers.cc:597] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:59.685477 11975 tablet_bootstrap.cc:492] T c69457ad1b93443c91f33a82ca91199b P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:59.685933 11975 ts_tablet_manager.cc:1403] T c69457ad1b93443c91f33a82ca91199b P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.075s user 0.020s sys 0.006s
I20260504 13:56:59.686025 9185 catalog_manager.cc:5671] T f337e9698fe64282a684e90d9c61da5a P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.688588 11975 raft_consensus.cc:359] T c69457ad1b93443c91f33a82ca91199b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.689236 11975 raft_consensus.cc:385] T c69457ad1b93443c91f33a82ca91199b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:59.689525 11975 raft_consensus.cc:740] T c69457ad1b93443c91f33a82ca91199b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:59.691825 9185 catalog_manager.cc:5671] T 57983fcecdb64a42884f797b4db57df0 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.694109 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
I20260504 13:56:59.695648 9185 catalog_manager.cc:5671] T 9f80ce19e8f9498b98952a371476f185 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:59.698799 9185 catalog_manager.cc:5671] T e29a7365caf04506b15089f10e6c1677 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
W20260504 13:56:59.699743 9767 consensus_peers.cc:597] T 88619eaa917d4c33a910fc1347ab126e P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:59.700294 9185 catalog_manager.cc:5671] T 03e1d998c4314db3b3682a0510393749 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.702325 9185 catalog_manager.cc:5671] T 6f4c8ffb2175493a981c18676485ac04 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.703689 9185 catalog_manager.cc:5671] T 23ff0d52a90c4056880190dbd936cb8e P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.705029 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 23ff0d52a90c4056880190dbd936cb8e (table default.Table_69c53f858e044915b4cc201c8450f4b6 [id=1d2bbdaf6382433cb8c2a55f62f0388d]). Expected version 1 got 0
I20260504 13:56:59.705636 9185 catalog_manager.cc:5671] T 75a43e11c8b34bd7bcddbf6e9b8f30dc P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.707650 9185 catalog_manager.cc:5671] T 644170f17f1b45a19677d958b9745e16 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.710942 9185 catalog_manager.cc:5671] T 99fe335b131b45a587ec1cf526dd7f79 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.712184 9185 catalog_manager.cc:5671] T bd635a863f7a41d087947f619e41f95c P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
W20260504 13:56:59.713510 9767 consensus_peers.cc:597] T 6323e1c10e7643f8a28f04e285f620e9 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:59.715024 9185 catalog_manager.cc:5671] T b7b942037c0e45a6b3cccba288b90b00 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.716791 9185 catalog_manager.cc:5671] T 92480654807d495b879576b0195871bd P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.717916 9185 catalog_manager.cc:5763] TS 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369) does not have the latest schema for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]). Expected version 1 got 0
I20260504 13:56:59.719099 9185 catalog_manager.cc:5671] T 564d0fafb49a4b4bb3bdb240c74c3410 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.722343 9185 catalog_manager.cc:5671] T 8d6db45d2e5c41a3b30cede3301694d3 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.724156 11975 consensus_queue.cc:260] T c69457ad1b93443c91f33a82ca91199b P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.725106 11969 raft_consensus.cc:493] T c69457ad1b93443c91f33a82ca91199b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:56:59.725574 11969 raft_consensus.cc:515] T c69457ad1b93443c91f33a82ca91199b P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.726642 11975 ts_tablet_manager.cc:1434] T c69457ad1b93443c91f33a82ca91199b P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s
I20260504 13:56:59.730482 11969 leader_election.cc:290] T c69457ad1b93443c91f33a82ca91199b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117), 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369)
I20260504 13:56:59.724224 9185 catalog_manager.cc:5671] T 6dd139c347f7406aac340cc6ac994273 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } }
I20260504 13:56:59.731978 9185 catalog_manager.cc:5671] T 2c09e9076214405eaa8846fe08158bb3 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.733261 9185 catalog_manager.cc:5671] T c1a01231bb044433ae5827d07a6592f8 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.735302 9185 catalog_manager.cc:5671] T eef975650610423c88ce8e4cfb0b013b P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:59.737565 9185 catalog_manager.cc:5671] T 6323e1c10e7643f8a28f04e285f620e9 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } }
I20260504 13:56:59.738943 9185 catalog_manager.cc:5671] T cf11f17e95dc411294c755e17c57811e P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.740402 9185 catalog_manager.cc:5671] T e9e66dacf5794b6195575556be889bce P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
W20260504 13:56:59.743567 9767 consensus_peers.cc:597] T b7b942037c0e45a6b3cccba288b90b00 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.744719 9767 consensus_peers.cc:597] T 3404ba04d3ec4a44a5e625536b941b58 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:59.741717 9185 catalog_manager.cc:5671] T af5c5a6cae4e4b6ea3eece2e21a587ec P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } }
I20260504 13:56:59.747336 9185 catalog_manager.cc:5671] T df71dc4f2cf84cda8cfc534a0eea86f8 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.748684 9185 catalog_manager.cc:5671] T 7989338bc4bf4bad8fb8fe37587d0a16 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.749341 11975 tablet_bootstrap.cc:492] T 0bb1a301d1504d89973e8b17c39c2e8a P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:59.749949 9185 catalog_manager.cc:5671] T 37b1664312d347288680d97b62c2aecb P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beceb6317444247b31b44a6581707c6 (127.8.238.194). New cstate: current_term: 1 leader_uuid: "5beceb6317444247b31b44a6581707c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } health_report { overall_health: HEALTHY } } }
I20260504 13:56:59.751541 9185 catalog_manager.cc:5671] T e0e70772b7ce4f27a941918aaf66a865 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.752753 9185 catalog_manager.cc:5671] T b84c3bedc7a84e0d8a873fb15689b314 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
I20260504 13:56:59.753983 9185 catalog_manager.cc:5671] T ddd30e9259a04d59b5c1bb82aabf7952 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4051d2bf0bf410eb8acd107560df888 (127.8.238.195). New cstate: current_term: 1 leader_uuid: "d4051d2bf0bf410eb8acd107560df888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
W20260504 13:56:59.755404 9767 leader_election.cc:336] T c69457ad1b93443c91f33a82ca91199b P d4051d2bf0bf410eb8acd107560df888 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111)
I20260504 13:56:59.755332 9185 catalog_manager.cc:5671] T 6677a93d01d54f15a506ae1474bcb501 P 5beceb6317444247b31b44a6581707c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193). New cstate: current_term: 1 leader_uuid: "7b79fe7dd8934dd996393a2d96eabf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } } }
W20260504 13:56:59.764515 9767 consensus_peers.cc:597] T f337e9698fe64282a684e90d9c61da5a P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:59.775274 11975 tablet_bootstrap.cc:654] T 0bb1a301d1504d89973e8b17c39c2e8a P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:59.796706 9176 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 23ff0d52a90c4056880190dbd936cb8e (table default.Table_69c53f858e044915b4cc201c8450f4b6 [id=1d2bbdaf6382433cb8c2a55f62f0388d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 23ff0d52a90c4056880190dbd936cb8e consensus state has no leader
I20260504 13:56:59.828987 11975 tablet_bootstrap.cc:492] T 0bb1a301d1504d89973e8b17c39c2e8a P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:59.829536 11975 ts_tablet_manager.cc:1403] T 0bb1a301d1504d89973e8b17c39c2e8a P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.080s user 0.011s sys 0.001s
I20260504 13:56:59.832232 11975 raft_consensus.cc:359] T 0bb1a301d1504d89973e8b17c39c2e8a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.833011 11975 raft_consensus.cc:385] T 0bb1a301d1504d89973e8b17c39c2e8a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:59.833343 11975 raft_consensus.cc:740] T 0bb1a301d1504d89973e8b17c39c2e8a P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:59.834040 11975 consensus_queue.cc:260] T 0bb1a301d1504d89973e8b17c39c2e8a P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.836788 11975 ts_tablet_manager.cc:1434] T 0bb1a301d1504d89973e8b17c39c2e8a P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260504 13:56:59.839511 11975 tablet_bootstrap.cc:492] T ba270b1227d14c60bd07046a73c6b7ec P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
I20260504 13:56:59.879824 11975 tablet_bootstrap.cc:654] T ba270b1227d14c60bd07046a73c6b7ec P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
I20260504 13:56:59.900079 11975 tablet_bootstrap.cc:492] T ba270b1227d14c60bd07046a73c6b7ec P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:59.900602 11975 ts_tablet_manager.cc:1403] T ba270b1227d14c60bd07046a73c6b7ec P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.005s
I20260504 13:56:59.903386 11975 raft_consensus.cc:359] T ba270b1227d14c60bd07046a73c6b7ec P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.905835 11975 raft_consensus.cc:385] T ba270b1227d14c60bd07046a73c6b7ec P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260504 13:56:59.906376 9767 consensus_peers.cc:597] T 2c09e9076214405eaa8846fe08158bb3 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:59.907523 11975 raft_consensus.cc:740] T ba270b1227d14c60bd07046a73c6b7ec P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:59.908303 11975 consensus_queue.cc:260] T ba270b1227d14c60bd07046a73c6b7ec P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.911000 11975 ts_tablet_manager.cc:1434] T ba270b1227d14c60bd07046a73c6b7ec P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.010s user 0.000s sys 0.005s
I20260504 13:56:59.924990 11975 tablet_bootstrap.cc:492] T a5f2bfe937b34c33bad8b736384248f9 P d4051d2bf0bf410eb8acd107560df888: Bootstrap starting.
W20260504 13:56:59.931625 9767 consensus_peers.cc:597] T 59f9115b286d44f0b7eac7c74275cb21 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.936457 9767 consensus_peers.cc:597] T 982f7c53a1f0482e862b53aeeee7cd25 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:56:59.946110 9767 consensus_peers.cc:597] T 99e819494171485e9af8bc8ec33a6ff0 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:59.946302 11975 tablet_bootstrap.cc:654] T a5f2bfe937b34c33bad8b736384248f9 P d4051d2bf0bf410eb8acd107560df888: Neither blocks nor log segments found. Creating new log.
W20260504 13:56:59.962662 9767 consensus_peers.cc:597] T 36c8e9ba29a1466082a3e8bedec87b91 P d4051d2bf0bf410eb8acd107560df888 -> Peer 7b79fe7dd8934dd996393a2d96eabf00 (127.8.238.193:33117): Couldn't send request to peer 7b79fe7dd8934dd996393a2d96eabf00. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.193:33117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:56:59.977175 11975 tablet_bootstrap.cc:492] T a5f2bfe937b34c33bad8b736384248f9 P d4051d2bf0bf410eb8acd107560df888: No bootstrap required, opened a new log
I20260504 13:56:59.977610 11975 ts_tablet_manager.cc:1403] T a5f2bfe937b34c33bad8b736384248f9 P d4051d2bf0bf410eb8acd107560df888: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.009s
I20260504 13:56:59.980130 11975 raft_consensus.cc:359] T a5f2bfe937b34c33bad8b736384248f9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.980770 11975 raft_consensus.cc:385] T a5f2bfe937b34c33bad8b736384248f9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:56:59.981060 11975 raft_consensus.cc:740] T a5f2bfe937b34c33bad8b736384248f9 P d4051d2bf0bf410eb8acd107560df888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4051d2bf0bf410eb8acd107560df888, State: Initialized, Role: FOLLOWER
I20260504 13:56:59.981822 11975 consensus_queue.cc:260] T a5f2bfe937b34c33bad8b736384248f9 P d4051d2bf0bf410eb8acd107560df888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b79fe7dd8934dd996393a2d96eabf00" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 33117 } } peers { permanent_uuid: "d4051d2bf0bf410eb8acd107560df888" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 34563 } } peers { permanent_uuid: "5beceb6317444247b31b44a6581707c6" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40369 } }
I20260504 13:56:59.983588 11975 ts_tablet_manager.cc:1434] T a5f2bfe937b34c33bad8b736384248f9 P d4051d2bf0bf410eb8acd107560df888: Time spent starting tablet: real 0.006s user 0.005s sys 0.003s
I20260504 13:57:00.005440 9181 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 57983fcecdb64a42884f797b4db57df0 (table default.Table_59e33df553a4469b8fd50d46d7ce9f97 [id=8501cecf36f94426bb887de3986dc537]). Expected version 1 got 0
W20260504 13:57:00.009362 10640 connection.cc:570] server connection from 127.8.238.194:56157 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:00.014855 9768 connection.cc:570] client connection to 127.8.238.194:40369 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:00.028151 11979 negotiation.cc:336] Failed RPC negotiation. Trace:
0504 13:57:00.010674 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.8.238.194:40369 (local address 127.0.0.1:51242)
0504 13:57:00.011388 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0504 13:57:00.011431 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0504 13:57:00.011837 (+ 406us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0504 13:57:00.027340 (+ 15503us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.238.194:40369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":69}
W20260504 13:57:00.030006 11982 negotiation.cc:336] Failed RPC negotiation. Trace:
0504 13:57:00.022631 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.8.238.194:40369 (local address 127.8.238.195:60077)
0504 13:57:00.023337 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0504 13:57:00.023380 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0504 13:57:00.023788 (+ 408us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0504 13:57:00.029118 (+ 5330us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.238.194:40369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":65}
I20260504 13:57:00.030001 9181 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 23ff0d52a90c4056880190dbd936cb8e (table default.Table_69c53f858e044915b4cc201c8450f4b6 [id=1d2bbdaf6382433cb8c2a55f62f0388d]). Expected version 1 got 0
W20260504 13:57:00.030726 9768 consensus_peers.cc:597] T 03e1d998c4314db3b3682a0510393749 P d4051d2bf0bf410eb8acd107560df888 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.194:40369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:57:00.031241 9768 consensus_peers.cc:597] T cf11f17e95dc411294c755e17c57811e P d4051d2bf0bf410eb8acd107560df888 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.194:40369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:57:00.031222 9181 catalog_manager.cc:5763] TS d4051d2bf0bf410eb8acd107560df888 (127.8.238.195:34563) does not have the latest schema for tablet 92480654807d495b879576b0195871bd (table default.Table_c4e8170ca1324d54b2bd104e88eaeb6a [id=e8c005ae044b43a3a7e92f96c53ebce1]). Expected version 1 got 0
W20260504 13:57:00.031651 9768 consensus_peers.cc:597] T e0e70772b7ce4f27a941918aaf66a865 P d4051d2bf0bf410eb8acd107560df888 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.194:40369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:57:00.031991 9768 consensus_peers.cc:597] T ddd30e9259a04d59b5c1bb82aabf7952 P d4051d2bf0bf410eb8acd107560df888 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.194:40369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:57:00.032343 9768 consensus_peers.cc:597] T acd5f7d3f7c545d18fe19b64e4bc1de6 P d4051d2bf0bf410eb8acd107560df888 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.194:40369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:57:00.032667 9768 consensus_peers.cc:597] T 57983fcecdb64a42884f797b4db57df0 P d4051d2bf0bf410eb8acd107560df888 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.194:40369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:57:00.032982 9768 consensus_peers.cc:597] T 8d6db45d2e5c41a3b30cede3301694d3 P d4051d2bf0bf410eb8acd107560df888 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.194:40369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:57:00.033363 9768 consensus_peers.cc:597] T b8ffeae4e5af4cc8a3eb3bc0bedf18cc P d4051d2bf0bf410eb8acd107560df888 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.194:40369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260504 13:57:00.034149 9768 consensus_peers.cc:597] T 7989338bc4bf4bad8fb8fe37587d0a16 P d4051d2bf0bf410eb8acd107560df888 -> Peer 5beceb6317444247b31b44a6581707c6 (127.8.238.194:40369): Couldn't send request to peer 5beceb6317444247b31b44a6581707c6. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.194:40369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:57:00.034493 9147 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu with pid 9735
I20260504 13:57:00.044111 9185 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: default.Table_d58041edcf8545f592c4a66547dd4a75 [id=3822050e2a9d4a9db8b0e36430fd0b7f] alter complete (version 3)
I20260504 13:57:00.059056 9185 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: default.Table_d32487a6e29d4021b8ea3bab67b31164 [id=a117c93d36da4d1aab22a554af5259d4] alter complete (version 1)
I20260504 13:57:00.082347 9185 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: default.Table_982e9b0871ac49a29936097dfa915691 [id=3814e135130a4213a46df3cd93eca49e] alter complete (version 2)
I20260504 13:57:00.107450 9185 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 806407748d564439807fe5d73db9abda: default.Table_270726cef63b42e39368cdf6aae3fe18 [id=3e8d8cb6a0b049f8843286a4421e121a] alter complete (version 1)
W20260504 13:57:00.123675 9174 connection.cc:463] server connection from 127.8.238.194:45997 torn down before Call kudu.master.MasterService.TSHeartbeat from 127.8.238.194:45997 (request call id 293) could send its response
W20260504 13:57:00.144719 9175 connection.cc:570] client connection to 127.8.238.195:34563 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:00.144989 9173 connection.cc:570] server connection from 127.8.238.195:33659 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260504 13:57:00.146741 9147 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu with pid 9156
I20260504 13:57:00.201160 9147 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu with pid 10086
I20260504 13:57:00.240687 9147 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu with pid 9301
2026-05-04T13:57:00Z chronyd exiting
I20260504 13:57:00.304117 9147 master-stress-test.cc:585] Tables created: 152
I20260504 13:57:00.304317 9147 master-stress-test.cc:586] Tables altered: 65
I20260504 13:57:00.304441 9147 master-stress-test.cc:587] Tables deleted: 50
I20260504 13:57:00.304574 9147 master-stress-test.cc:588] Tablets replaced: 21
I20260504 13:57:00.304684 9147 master-stress-test.cc:589] Masters restarted: 1
[ OK ] CatalogManagerConfigurations/MasterStressTest.Test/0 (67478 ms)
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/3
2026-05-04T13:57:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-04T13:57:00Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskcMKItV/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskcMKItV/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/security.properties
2026-05-04 13:57:07: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskcMKItV/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskcMKItV/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260504 13:57:15.752992 9147 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
/tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.254:38403
--webserver_interface=127.8.238.254
--webserver_port=0
--builtin_ntp_servers=127.8.238.212:41493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--hive_metastore_uris=thrift://127.0.0.1:39899
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260504 13:57:16.062284 12277 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260504 13:57:16.062906 12277 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260504 13:57:16.063364 12277 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260504 13:57:16.095348 12277 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260504 13:57:16.095613 12277 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260504 13:57:16.095928 12277 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260504 13:57:16.096189 12277 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260504 13:57:16.096457 12277 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260504 13:57:16.096732 12277 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260504 13:57:16.135349 12277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.238.212:41493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:39899
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.254:38403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.238.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 04 May 2026 13:43:23 UTC on e7f111948823
build id 11743
TSAN enabled
I20260504 13:57:16.136803 12277 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 13:57:16.138406 12277 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 13:57:16.150019 12284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:57:16.150071 12283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:57:17.368870 12277 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.218s user 0.387s sys 0.829s
W20260504 13:57:17.369374 12277 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.219s user 0.387s sys 0.830s
W20260504 13:57:17.370050 12285 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1214 milliseconds
W20260504 13:57:17.371207 12287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:57:17.371287 12277 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260504 13:57:17.372542 12277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260504 13:57:17.375288 12277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260504 13:57:17.376675 12277 hybrid_clock.cc:648] HybridClock initialized: now 1777903037376623 us; error 62 us; skew 500 ppm
I20260504 13:57:17.377461 12277 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 13:57:17.385277 12277 webserver.cc:492] Webserver started at http://127.8.238.254:39275/ using document root <none> and password file <none>
I20260504 13:57:17.386226 12277 fs_manager.cc:362] Metadata directory not provided
I20260504 13:57:17.386487 12277 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 13:57:17.386940 12277 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 13:57:17.392822 12277 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/data/instance:
uuid: "bc3d546912c9427aaca93bdafc33dd6d"
format_stamp: "Formatted at 2026-05-04 13:57:17 on dist-test-slave-f9wt"
I20260504 13:57:17.393949 12277 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/wal/instance:
uuid: "bc3d546912c9427aaca93bdafc33dd6d"
format_stamp: "Formatted at 2026-05-04 13:57:17 on dist-test-slave-f9wt"
I20260504 13:57:17.402205 12277 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20260504 13:57:17.408594 12293 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 13:57:17.409919 12277 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260504 13:57:17.410264 12277 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/data,/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/wal
uuid: "bc3d546912c9427aaca93bdafc33dd6d"
format_stamp: "Formatted at 2026-05-04 13:57:17 on dist-test-slave-f9wt"
I20260504 13:57:17.410641 12277 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 13:57:17.494704 12277 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 13:57:17.496361 12277 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 13:57:17.496842 12277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 13:57:17.566351 12277 rpc_server.cc:307] RPC server started. Bound to: 127.8.238.254:38403
I20260504 13:57:17.566473 12344 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.238.254:38403 every 8 connection(s)
I20260504 13:57:17.569092 12277 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/data/info.pb
I20260504 13:57:17.573137 9147 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu as pid 12277
I20260504 13:57:17.573565 9147 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-0/wal/instance
I20260504 13:57:17.574641 12345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:17.576100 9147 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
/tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.253:35717
--webserver_interface=127.8.238.253
--webserver_port=0
--builtin_ntp_servers=127.8.238.212:41493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--hive_metastore_uris=thrift://127.0.0.1:39899
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260504 13:57:17.584782 12345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } has no permanent_uuid. Determining permanent_uuid...
I20260504 13:57:17.610436 12345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } has no permanent_uuid. Determining permanent_uuid...
W20260504 13:57:17.612653 12296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
W20260504 13:57:17.616369 12345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.253:35717: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:57:17.665841 12345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } attempt: 1
W20260504 13:57:17.669999 12345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.253:35717: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:57:17.738445 12345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } attempt: 2
W20260504 13:57:17.742486 12345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.253:35717: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:57:17.833885 12345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } attempt: 3
W20260504 13:57:17.837795 12345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.253:35717: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
W20260504 13:57:17.886453 12346 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260504 13:57:17.887036 12346 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260504 13:57:17.887493 12346 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260504 13:57:17.920384 12346 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260504 13:57:17.920689 12346 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260504 13:57:17.920953 12346 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260504 13:57:17.921217 12346 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260504 13:57:17.921473 12346 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260504 13:57:17.921722 12346 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260504 13:57:17.958851 12346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.238.212:41493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:39899
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.253:35717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.238.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 04 May 2026 13:43:23 UTC on e7f111948823
build id 11743
TSAN enabled
I20260504 13:57:17.960289 12346 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 13:57:17.961903 12346 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 13:57:17.973057 12355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:57:17.981178 12345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } attempt: 4
W20260504 13:57:17.985092 12345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.253:35717: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:57:18.284538 12345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } attempt: 5
W20260504 13:57:18.290797 12345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.253:35717: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:57:18.838425 12345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } attempt: 6
W20260504 13:57:18.853016 12345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.253:35717: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
W20260504 13:57:17.974305 12356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:57:19.218494 12358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:57:19.222431 12357 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1247 milliseconds
I20260504 13:57:19.222512 12346 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260504 13:57:19.223832 12346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260504 13:57:19.226974 12346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260504 13:57:19.228440 12346 hybrid_clock.cc:648] HybridClock initialized: now 1777903039228400 us; error 65 us; skew 500 ppm
I20260504 13:57:19.229266 12346 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 13:57:19.235395 12346 webserver.cc:492] Webserver started at http://127.8.238.253:44299/ using document root <none> and password file <none>
I20260504 13:57:19.236357 12346 fs_manager.cc:362] Metadata directory not provided
I20260504 13:57:19.236604 12346 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 13:57:19.237066 12346 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 13:57:19.241390 12346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/data/instance:
uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4"
format_stamp: "Formatted at 2026-05-04 13:57:19 on dist-test-slave-f9wt"
I20260504 13:57:19.242496 12346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/wal/instance:
uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4"
format_stamp: "Formatted at 2026-05-04 13:57:19 on dist-test-slave-f9wt"
I20260504 13:57:19.250221 12346 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260504 13:57:19.256014 12367 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 13:57:19.257067 12346 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260504 13:57:19.257414 12346 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/data,/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/wal
uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4"
format_stamp: "Formatted at 2026-05-04 13:57:19 on dist-test-slave-f9wt"
I20260504 13:57:19.257748 12346 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 13:57:19.311936 12346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 13:57:19.313424 12346 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 13:57:19.313884 12346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 13:57:19.384569 12346 rpc_server.cc:307] RPC server started. Bound to: 127.8.238.253:35717
I20260504 13:57:19.384649 12418 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.238.253:35717 every 8 connection(s)
I20260504 13:57:19.387365 12346 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/data/info.pb
I20260504 13:57:19.388012 9147 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu as pid 12346
I20260504 13:57:19.388521 9147 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-1/wal/instance
I20260504 13:57:19.391466 9147 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
/tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.252:35825
--webserver_interface=127.8.238.252
--webserver_port=0
--builtin_ntp_servers=127.8.238.212:41493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--hive_metastore_uris=thrift://127.0.0.1:39899
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260504 13:57:19.393436 12419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:19.408514 12419 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } has no permanent_uuid. Determining permanent_uuid...
I20260504 13:57:19.433908 12419 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } has no permanent_uuid. Determining permanent_uuid...
I20260504 13:57:19.448540 12419 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } has no permanent_uuid. Determining permanent_uuid...
W20260504 13:57:19.451231 12371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
W20260504 13:57:19.455988 12419 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:35825: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:19.505518 12419 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } attempt: 1
W20260504 13:57:19.509791 12419 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:35825: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:19.578219 12419 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } attempt: 2
W20260504 13:57:19.583024 12419 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:35825: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:19.674527 12419 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } attempt: 3
W20260504 13:57:19.678571 12419 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:35825: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
W20260504 13:57:19.710901 12420 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260504 13:57:19.711484 12420 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260504 13:57:19.711846 12420 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260504 13:57:19.743702 12420 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260504 13:57:19.743942 12420 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260504 13:57:19.744139 12420 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260504 13:57:19.744335 12420 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260504 13:57:19.744539 12420 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260504 13:57:19.744729 12420 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260504 13:57:19.781016 12420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.238.212:41493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:39899
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.252:35825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.238.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 04 May 2026 13:43:23 UTC on e7f111948823
build id 11743
TSAN enabled
I20260504 13:57:19.782579 12420 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 13:57:19.784109 12420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 13:57:19.793968 12430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:57:19.821997 12419 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } attempt: 4
W20260504 13:57:19.825708 12419 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:35825: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:19.913635 12345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } attempt: 7
I20260504 13:57:19.933570 12345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } has no permanent_uuid. Determining permanent_uuid...
W20260504 13:57:19.942698 12345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:35825: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:20.001266 12345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } attempt: 1
W20260504 13:57:20.007863 12345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:35825: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:20.089594 12345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } attempt: 2
W20260504 13:57:20.097477 12345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:35825: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:20.125164 12419 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } attempt: 5
W20260504 13:57:20.132223 12419 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:35825: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:20.183030 12345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } attempt: 3
W20260504 13:57:20.189263 12345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:35825: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:20.329818 12345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } attempt: 4
W20260504 13:57:20.336369 12345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:35825: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:20.619974 12345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } attempt: 5
W20260504 13:57:20.626627 12345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:35825: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:20.679899 12419 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } attempt: 6
W20260504 13:57:20.690757 12419 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:35825: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
W20260504 13:57:19.795567 12431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:57:21.058029 12433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:57:21.059796 12432 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1260 milliseconds
I20260504 13:57:21.059881 12420 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260504 13:57:21.061112 12420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260504 13:57:21.064175 12420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260504 13:57:21.065629 12420 hybrid_clock.cc:648] HybridClock initialized: now 1777903041065588 us; error 61 us; skew 500 ppm
I20260504 13:57:21.066583 12420 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 13:57:21.072952 12420 webserver.cc:492] Webserver started at http://127.8.238.252:35067/ using document root <none> and password file <none>
I20260504 13:57:21.073882 12420 fs_manager.cc:362] Metadata directory not provided
I20260504 13:57:21.074110 12420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 13:57:21.074545 12420 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 13:57:21.078548 12420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data/instance:
uuid: "26e2451840464331a15d5291c3f09034"
format_stamp: "Formatted at 2026-05-04 13:57:21 on dist-test-slave-f9wt"
I20260504 13:57:21.079629 12420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal/instance:
uuid: "26e2451840464331a15d5291c3f09034"
format_stamp: "Formatted at 2026-05-04 13:57:21 on dist-test-slave-f9wt"
I20260504 13:57:21.086458 12420 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260504 13:57:21.091773 12444 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 13:57:21.092967 12420 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260504 13:57:21.093283 12420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data,/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal
uuid: "26e2451840464331a15d5291c3f09034"
format_stamp: "Formatted at 2026-05-04 13:57:21 on dist-test-slave-f9wt"
I20260504 13:57:21.093611 12420 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 13:57:21.154214 12420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 13:57:21.155732 12420 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 13:57:21.156180 12420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 13:57:21.179255 12345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } attempt: 6
W20260504 13:57:21.187290 12345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.8.238.252:35825: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:21.233214 12420 rpc_server.cc:307] RPC server started. Bound to: 127.8.238.252:35825
I20260504 13:57:21.233294 12497 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.238.252:35825 every 8 connection(s)
I20260504 13:57:21.235939 12420 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data/info.pb
I20260504 13:57:21.241250 12498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:21.245839 9147 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu as pid 12420
I20260504 13:57:21.246258 9147 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal/instance
I20260504 13:57:21.254606 12498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } has no permanent_uuid. Determining permanent_uuid...
I20260504 13:57:21.278137 12498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } has no permanent_uuid. Determining permanent_uuid...
I20260504 13:57:21.287434 12498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } has no permanent_uuid. Determining permanent_uuid...
I20260504 13:57:21.302974 12498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: Bootstrap starting.
I20260504 13:57:21.309211 12498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:21.310920 12498 log.cc:826] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: Log is configured to *not* fsync() on all Append() calls
I20260504 13:57:21.315042 12498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: No bootstrap required, opened a new log
I20260504 13:57:21.334275 12498 raft_consensus.cc:359] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:21.334937 12498 raft_consensus.cc:385] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:21.335249 12498 raft_consensus.cc:740] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e2451840464331a15d5291c3f09034, State: Initialized, Role: FOLLOWER
I20260504 13:57:21.336125 12498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:21.339003 12506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:21.339689 12506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:57:21.340641 12498 sys_catalog.cc:565] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: configured and running, proceeding with master startup.
W20260504 13:57:21.389838 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.390190 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.412014 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.412331 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.412861 12450 tablet.cc:2404] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260504 13:57:21.367883 12420 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: OK, instance_id { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" instance_seqno: 1777903037547390 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260504 13:57:21.422842 12420 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 13:57:21.426946 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.427302 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.441416 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.441737 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.455988 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.456359 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.470513 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.470880 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.484928 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.485298 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.499444 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.499752 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.512826 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.513070 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.527281 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.527690 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.540617 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.540869 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.555152 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.555487 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.568666 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.568941 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.581782 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.582031 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.595099 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.595415 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260504 13:57:21.596689 12506 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:21.597123 12506 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:21.599417 12506 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:57:21.609392 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.609714 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.623848 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.624236 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.639550 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.639873 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.653954 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.654315 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.668406 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.668711 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.682734 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.683053 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.697103 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.697456 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.713929 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.714246 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.727042 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.727423 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.742707 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.743036 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260504 13:57:21.751338 12419 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } attempt: 7
W20260504 13:57:21.757211 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.757525 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.771617 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.771906 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260504 13:57:21.780092 12419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4: Bootstrap starting.
W20260504 13:57:21.785849 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.786123 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260504 13:57:21.786356 12419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:21.788619 12419 log.cc:826] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4: Log is configured to *not* fsync() on all Append() calls
I20260504 13:57:21.793218 12419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4: No bootstrap required, opened a new log
W20260504 13:57:21.799739 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.800071 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260504 13:57:21.813632 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.813908 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260504 13:57:21.819593 12419 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:21.820406 12419 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:21.820693 12419 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc832d2f7f748dbb8e3ba2f73ec98a4, State: Initialized, Role: FOLLOWER
I20260504 13:57:21.821547 12419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:21.824223 12526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:21.825055 12526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:57:21.825951 12419 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260504 13:57:21.827478 12394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26e2451840464331a15d5291c3f09034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" is_pre_election: true
W20260504 13:57:21.827666 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.828183 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260504 13:57:21.828516 12394 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e2451840464331a15d5291c3f09034 in term 0.
I20260504 13:57:21.830127 12447 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e2451840464331a15d5291c3f09034, 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters:
I20260504 13:57:21.831068 12506 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:57:21.831426 12506 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:57:21.831746 12506 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:57:21.838457 12506 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:21.848682 12506 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 1 election: Requested vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
I20260504 13:57:21.850528 12394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26e2451840464331a15d5291c3f09034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4"
I20260504 13:57:21.851317 12394 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:57:21.855722 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.859592 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260504 13:57:21.862772 12394 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e2451840464331a15d5291c3f09034 in term 1.
I20260504 13:57:21.864094 12447 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e2451840464331a15d5291c3f09034, 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters:
I20260504 13:57:21.864866 12506 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:57:21.866762 12506 raft_consensus.cc:697] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 1 LEADER]: Becoming Leader. State: Replica: 26e2451840464331a15d5291c3f09034, State: Running, Role: LEADER
I20260504 13:57:21.867725 12506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:21.885643 12537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26e2451840464331a15d5291c3f09034. Latest consensus state: current_term: 1 leader_uuid: "26e2451840464331a15d5291c3f09034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:21.886363 12537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: This master's current role is: LEADER
I20260504 13:57:21.889156 12543 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260504 13:57:21.890941 12520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.899298 12520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260504 13:57:21.907506 12544 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260504 13:57:21.915962 12544 catalog_manager.cc:1494] Initializing Kudu cluster ID...
W20260504 13:57:21.935307 12543 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.936058 12543 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260504 13:57:21.941938 12394 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Refusing update from remote peer 26e2451840464331a15d5291c3f09034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 13:57:21.943596 12537 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:57:21.900544 12346 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: OK, instance_id { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" instance_seqno: 1777903037547390 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260504 13:57:21.956627 12346 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:57:21.987304 12526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26e2451840464331a15d5291c3f09034. Latest consensus state: current_term: 1 leader_uuid: "26e2451840464331a15d5291c3f09034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:21.991422 12506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26e2451840464331a15d5291c3f09034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:21.992354 12506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: This master's current role is: LEADER
I20260504 13:57:21.993187 12544 catalog_manager.cc:1357] Generated new cluster ID: 8a9361cd032446ef9af554d6cfd5f3e8
I20260504 13:57:21.993539 12544 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260504 13:57:21.993113 12526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [sys.catalog]: This master's current role is: FOLLOWER
W20260504 13:57:21.994451 12543 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260504 13:57:21.996301 12543 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260504 13:57:22.019380 12526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26e2451840464331a15d5291c3f09034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:22.023739 12526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:57:22.038026 12543 catalog_manager.cc:1269] Loaded cluster ID: 8a9361cd032446ef9af554d6cfd5f3e8
I20260504 13:57:22.039055 12543 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4: loading cluster ID for follower catalog manager: success
I20260504 13:57:22.044042 12544 catalog_manager.cc:1380] Generated new certificate authority record
I20260504 13:57:22.046126 12544 catalog_manager.cc:1514] Loading token signing keys...
W20260504 13:57:22.060894 12543 catalog_manager.cc:1586] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260504 13:57:22.061888 12543 catalog_manager.cc:883] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260504 13:57:22.089288 12544 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: Generated new TSK 0
I20260504 13:57:22.090389 12544 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260504 13:57:22.094839 12544 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260504 13:57:22.099011 12544 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260504 13:57:22.104948 12543 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4: acquiring CA information for follower catalog manager: success
I20260504 13:57:22.121914 12543 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260504 13:57:22.146306 9147 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
/tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.238.193:0
--local_ip_for_outbound_sockets=127.8.238.193
--webserver_interface=127.8.238.193
--webserver_port=0
--tserver_master_addrs=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--builtin_ntp_servers=127.8.238.212:41493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20260504 13:57:22.198334 12447 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.238.254:38403 timed out after 0.598s (SENT)
I20260504 13:57:22.220830 12345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } attempt: 7
I20260504 13:57:22.266937 12345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d: Bootstrap starting.
I20260504 13:57:22.276494 12345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:22.278771 12345 log.cc:826] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d: Log is configured to *not* fsync() on all Append() calls
I20260504 13:57:22.289994 12345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d: No bootstrap required, opened a new log
I20260504 13:57:22.320745 12345 raft_consensus.cc:359] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:22.321671 12345 raft_consensus.cc:385] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:22.322005 12345 raft_consensus.cc:740] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc3d546912c9427aaca93bdafc33dd6d, State: Initialized, Role: FOLLOWER
I20260504 13:57:22.323056 12345 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:22.347932 12557 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:22.349684 12557 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:57:22.350474 12557 raft_consensus.cc:493] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:22.355660 12317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26e2451840464331a15d5291c3f09034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc3d546912c9427aaca93bdafc33dd6d" is_pre_election: true
I20260504 13:57:22.356037 12320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26e2451840464331a15d5291c3f09034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc3d546912c9427aaca93bdafc33dd6d" is_pre_election: true
W20260504 13:57:22.357143 12320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59590 (request call id 2) took 757 ms (client timeout 598 ms). Trace:
W20260504 13:57:22.357529 12320 rpcz_store.cc:269] 0504 13:57:21.599995 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:57:21.600115 (+ 120us) service_pool.cc:224] Handling call
0504 13:57:22.357072 (+756957us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:57:22.358479 12319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26e2451840464331a15d5291c3f09034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc3d546912c9427aaca93bdafc33dd6d"
I20260504 13:57:22.359007 12345 sys_catalog.cc:565] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [sys.catalog]: configured and running, proceeding with master startup.
W20260504 13:57:22.359268 12319 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59590 (request call id 3) took 508 ms (client timeout 605 ms). Trace:
I20260504 13:57:22.359587 12557 raft_consensus.cc:515] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:22.366170 12557 leader_election.cc:290] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:22.368177 12473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3d546912c9427aaca93bdafc33dd6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034" is_pre_election: true
I20260504 13:57:22.370644 12318 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:57:22.359599 12319 rpcz_store.cc:269] 0504 13:57:21.850519 (+ 0us) service_pool.cc:167] Inserting onto call queue
0504 13:57:21.851834 (+ 1315us) service_pool.cc:224] Handling call
0504 13:57:22.359219 (+507385us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260504 13:57:22.376556 12393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3d546912c9427aaca93bdafc33dd6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" is_pre_election: true
I20260504 13:57:22.377954 12296 leader_election.cc:304] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc3d546912c9427aaca93bdafc33dd6d; no voters: 26e2451840464331a15d5291c3f09034, 6dc832d2f7f748dbb8e3ba2f73ec98a4
I20260504 13:57:22.382530 12318 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Refusing update from remote peer 26e2451840464331a15d5291c3f09034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260504 13:57:22.387715 12557 raft_consensus.cc:2749] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260504 13:57:22.406597 12537 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:57:22.484148 12557 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26e2451840464331a15d5291c3f09034. Latest consensus state: current_term: 1 leader_uuid: "26e2451840464331a15d5291c3f09034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:22.484848 12557 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:57:22.485883 12556 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26e2451840464331a15d5291c3f09034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:22.486665 12556 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:57:22.604092 12577 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260504 13:57:22.643193 12577 catalog_manager.cc:1269] Loaded cluster ID: 8a9361cd032446ef9af554d6cfd5f3e8
I20260504 13:57:22.643463 12577 catalog_manager.cc:1562] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d: loading cluster ID for follower catalog manager: success
I20260504 13:57:22.659202 12577 catalog_manager.cc:1584] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d: acquiring CA information for follower catalog manager: success
I20260504 13:57:22.678763 12577 catalog_manager.cc:1612] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260504 13:57:22.868606 12537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26e2451840464331a15d5291c3f09034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:22.870769 12537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: This master's current role is: LEADER
W20260504 13:57:23.046911 12552 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260504 13:57:23.047678 12552 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260504 13:57:23.048622 12552 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260504 13:57:23.092568 12586 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:23.096962 12586 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:23.104385 12318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bc3d546912c9427aaca93bdafc33dd6d" is_pre_election: true
I20260504 13:57:23.107542 12473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034" is_pre_election: true
I20260504 13:57:23.108058 12586 leader_election.cc:290] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:23.108979 12371 leader_election.cc:304] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters: 26e2451840464331a15d5291c3f09034, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:23.115841 12586 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:57:23.154722 12552 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260504 13:57:23.156420 12552 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.238.193
I20260504 13:57:23.231434 12552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.238.212:41493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.238.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.238.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.238.193
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 04 May 2026 13:43:23 UTC on e7f111948823
build id 11743
TSAN enabled
I20260504 13:57:23.233381 12552 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 13:57:23.235944 12552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 13:57:23.272675 12590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:57:23.668615 12596 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:23.670632 12596 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:23.675300 12318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bc3d546912c9427aaca93bdafc33dd6d" is_pre_election: true
I20260504 13:57:23.680120 12596 leader_election.cc:290] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:23.680291 12473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034" is_pre_election: true
I20260504 13:57:23.682179 12371 leader_election.cc:304] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters: 26e2451840464331a15d5291c3f09034, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:23.683959 12596 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:57:23.895618 12597 raft_consensus.cc:493] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:23.896421 12597 raft_consensus.cc:515] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:23.899330 12393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3d546912c9427aaca93bdafc33dd6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" is_pre_election: true
I20260504 13:57:23.904501 12473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3d546912c9427aaca93bdafc33dd6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034" is_pre_election: true
I20260504 13:57:23.906966 12297 leader_election.cc:304] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc3d546912c9427aaca93bdafc33dd6d; no voters: 26e2451840464331a15d5291c3f09034, 6dc832d2f7f748dbb8e3ba2f73ec98a4
I20260504 13:57:23.919227 12597 leader_election.cc:290] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:23.920921 12598 raft_consensus.cc:2749] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:57:24.479897 12599 raft_consensus.cc:493] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:24.480489 12599 raft_consensus.cc:515] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:24.492913 12599 leader_election.cc:290] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:24.494254 12473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3d546912c9427aaca93bdafc33dd6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034" is_pre_election: true
I20260504 13:57:24.496987 12393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3d546912c9427aaca93bdafc33dd6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" is_pre_election: true
I20260504 13:57:24.500049 12296 leader_election.cc:304] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc3d546912c9427aaca93bdafc33dd6d; no voters: 26e2451840464331a15d5291c3f09034, 6dc832d2f7f748dbb8e3ba2f73ec98a4
I20260504 13:57:24.501055 12599 raft_consensus.cc:2749] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:57:25.199954 12600 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:25.200520 12600 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:25.203387 12600 leader_election.cc:290] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:25.203410 12318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bc3d546912c9427aaca93bdafc33dd6d" is_pre_election: true
I20260504 13:57:25.207463 12473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034" is_pre_election: true
I20260504 13:57:25.210208 12371 leader_election.cc:304] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters: 26e2451840464331a15d5291c3f09034, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:25.211344 12600 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:57:25.772284 12601 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:25.773167 12601 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:25.776686 12318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bc3d546912c9427aaca93bdafc33dd6d" is_pre_election: true
I20260504 13:57:25.780470 12473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034" is_pre_election: true
I20260504 13:57:25.782893 12371 leader_election.cc:304] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters: 26e2451840464331a15d5291c3f09034, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:25.788229 12601 leader_election.cc:290] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:25.789309 12601 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:57:23.283776 12589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:57:23.286630 12593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:57:25.804296 12591 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2523 milliseconds
I20260504 13:57:25.804323 12552 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260504 13:57:25.809448 12552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260504 13:57:25.812139 12552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260504 13:57:25.813586 12552 hybrid_clock.cc:648] HybridClock initialized: now 1777903045813506 us; error 87 us; skew 500 ppm
I20260504 13:57:25.814633 12552 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 13:57:25.821864 12552 webserver.cc:492] Webserver started at http://127.8.238.193:33571/ using document root <none> and password file <none>
I20260504 13:57:25.823058 12552 fs_manager.cc:362] Metadata directory not provided
I20260504 13:57:25.823343 12552 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 13:57:25.823906 12552 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 13:57:25.830037 12552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/data/instance:
uuid: "910fbf5d51f843609f5caf25e52beae6"
format_stamp: "Formatted at 2026-05-04 13:57:25 on dist-test-slave-f9wt"
I20260504 13:57:25.831568 12552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/wal/instance:
uuid: "910fbf5d51f843609f5caf25e52beae6"
format_stamp: "Formatted at 2026-05-04 13:57:25 on dist-test-slave-f9wt"
I20260504 13:57:25.840494 12552 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260504 13:57:25.847527 12607 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 13:57:25.848706 12552 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260504 13:57:25.849090 12552 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/wal
uuid: "910fbf5d51f843609f5caf25e52beae6"
format_stamp: "Formatted at 2026-05-04 13:57:25 on dist-test-slave-f9wt"
I20260504 13:57:25.849411 12552 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 13:57:25.901117 12552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 13:57:25.902683 12552 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 13:57:25.903134 12552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 13:57:25.906396 12552 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 13:57:25.910815 12552 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 13:57:25.910997 12552 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 13:57:25.911377 12552 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 13:57:25.911583 12552 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260504 13:57:26.056800 12552 rpc_server.cc:307] RPC server started. Bound to: 127.8.238.193:35229
I20260504 13:57:26.056939 12720 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.238.193:35229 every 8 connection(s)
I20260504 13:57:26.059406 12552 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/data/info.pb
I20260504 13:57:26.068027 9147 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu as pid 12552
I20260504 13:57:26.068879 9147 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/wal/instance
I20260504 13:57:26.089339 9147 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
/tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.238.194:0
--local_ip_for_outbound_sockets=127.8.238.194
--webserver_interface=127.8.238.194
--webserver_port=0
--tserver_master_addrs=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--builtin_ntp_servers=127.8.238.212:41493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260504 13:57:26.105441 12721 heartbeater.cc:344] Connected to a master server at 127.8.238.252:35825
I20260504 13:57:26.105811 12723 heartbeater.cc:344] Connected to a master server at 127.8.238.254:38403
I20260504 13:57:26.106012 12721 heartbeater.cc:461] Registering TS with master...
I20260504 13:57:26.106329 12723 heartbeater.cc:461] Registering TS with master...
I20260504 13:57:26.106928 12722 heartbeater.cc:344] Connected to a master server at 127.8.238.253:35717
I20260504 13:57:26.107321 12722 heartbeater.cc:461] Registering TS with master...
I20260504 13:57:26.107570 12721 heartbeater.cc:507] Master 127.8.238.252:35825 requested a full tablet report, sending...
I20260504 13:57:26.107712 12723 heartbeater.cc:507] Master 127.8.238.254:38403 requested a full tablet report, sending...
I20260504 13:57:26.108179 12722 heartbeater.cc:507] Master 127.8.238.253:35717 requested a full tablet report, sending...
I20260504 13:57:26.110787 12463 ts_manager.cc:194] Registered new tserver with Master: 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229)
I20260504 13:57:26.111253 12384 ts_manager.cc:194] Registered new tserver with Master: 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229)
I20260504 13:57:26.111450 12310 ts_manager.cc:194] Registered new tserver with Master: 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229)
I20260504 13:57:26.113507 12463 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.238.193:54673
I20260504 13:57:26.183830 12732 raft_consensus.cc:493] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:26.184494 12732 raft_consensus.cc:515] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:26.186631 12732 leader_election.cc:290] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:26.187839 12393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3d546912c9427aaca93bdafc33dd6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" is_pre_election: true
I20260504 13:57:26.188673 12473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3d546912c9427aaca93bdafc33dd6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034" is_pre_election: true
I20260504 13:57:26.190762 12297 leader_election.cc:304] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc3d546912c9427aaca93bdafc33dd6d; no voters: 26e2451840464331a15d5291c3f09034, 6dc832d2f7f748dbb8e3ba2f73ec98a4
I20260504 13:57:26.191684 12732 raft_consensus.cc:2749] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:57:26.417850 12731 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260504 13:57:26.418354 12731 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260504 13:57:26.418929 12731 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260504 13:57:26.455790 12731 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260504 13:57:26.456736 12731 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.238.194
I20260504 13:57:26.494831 12731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.238.212:41493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.238.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.238.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.238.194
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 04 May 2026 13:43:23 UTC on e7f111948823
build id 11743
TSAN enabled
I20260504 13:57:26.496131 12731 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 13:57:26.497625 12731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 13:57:26.510605 12739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:57:26.616914 12721 heartbeater.cc:499] Master 127.8.238.252:35825 was elected leader, sending a full tablet report...
I20260504 13:57:27.785919 12744 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:27.786562 12744 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:27.789229 12744 leader_election.cc:290] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:27.790004 12318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bc3d546912c9427aaca93bdafc33dd6d" is_pre_election: true
I20260504 13:57:27.791361 12473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034" is_pre_election: true
I20260504 13:57:27.793073 12371 leader_election.cc:304] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters: 26e2451840464331a15d5291c3f09034, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:27.794039 12744 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:57:26.510958 12738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:57:27.863044 12741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:57:27.865886 12731 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260504 13:57:27.865828 12740 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1350 milliseconds
I20260504 13:57:27.867635 12731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260504 13:57:27.870389 12731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260504 13:57:27.871888 12731 hybrid_clock.cc:648] HybridClock initialized: now 1777903047871852 us; error 42 us; skew 500 ppm
I20260504 13:57:27.873026 12731 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 13:57:27.881124 12731 webserver.cc:492] Webserver started at http://127.8.238.194:36859/ using document root <none> and password file <none>
I20260504 13:57:27.882937 12731 fs_manager.cc:362] Metadata directory not provided
I20260504 13:57:27.883236 12731 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 13:57:27.883771 12731 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 13:57:27.888490 12731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/data/instance:
uuid: "3517f3498e5e4e9cb5cbfa239e4999e3"
format_stamp: "Formatted at 2026-05-04 13:57:27 on dist-test-slave-f9wt"
I20260504 13:57:27.889550 12731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/wal/instance:
uuid: "3517f3498e5e4e9cb5cbfa239e4999e3"
format_stamp: "Formatted at 2026-05-04 13:57:27 on dist-test-slave-f9wt"
I20260504 13:57:27.896538 12731 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260504 13:57:27.902086 12750 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 13:57:27.903117 12731 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260504 13:57:27.903532 12731 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/wal
uuid: "3517f3498e5e4e9cb5cbfa239e4999e3"
format_stamp: "Formatted at 2026-05-04 13:57:27 on dist-test-slave-f9wt"
I20260504 13:57:27.903864 12731 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 13:57:27.955578 12731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 13:57:27.957057 12731 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 13:57:27.957559 12731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 13:57:27.960150 12731 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 13:57:27.964731 12731 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 13:57:27.964928 12731 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 13:57:27.965126 12731 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 13:57:27.965257 12731 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260504 13:57:28.109326 12731 rpc_server.cc:307] RPC server started. Bound to: 127.8.238.194:40331
I20260504 13:57:28.109431 12862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.238.194:40331 every 8 connection(s)
I20260504 13:57:28.111975 12731 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/data/info.pb
I20260504 13:57:28.121181 9147 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu as pid 12731
I20260504 13:57:28.121881 9147 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/wal/instance
I20260504 13:57:28.156407 9147 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
/tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.238.195:0
--local_ip_for_outbound_sockets=127.8.238.195
--webserver_interface=127.8.238.195
--webserver_port=0
--tserver_master_addrs=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--builtin_ntp_servers=127.8.238.212:41493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260504 13:57:28.180030 12864 heartbeater.cc:344] Connected to a master server at 127.8.238.253:35717
I20260504 13:57:28.180215 12863 heartbeater.cc:344] Connected to a master server at 127.8.238.252:35825
I20260504 13:57:28.180610 12864 heartbeater.cc:461] Registering TS with master...
I20260504 13:57:28.181048 12863 heartbeater.cc:461] Registering TS with master...
I20260504 13:57:28.181833 12864 heartbeater.cc:507] Master 127.8.238.253:35717 requested a full tablet report, sending...
I20260504 13:57:28.182186 12863 heartbeater.cc:507] Master 127.8.238.252:35825 requested a full tablet report, sending...
I20260504 13:57:28.184031 12865 heartbeater.cc:344] Connected to a master server at 127.8.238.254:38403
I20260504 13:57:28.184355 12865 heartbeater.cc:461] Registering TS with master...
I20260504 13:57:28.184979 12383 ts_manager.cc:194] Registered new tserver with Master: 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331)
I20260504 13:57:28.185238 12865 heartbeater.cc:507] Master 127.8.238.254:38403 requested a full tablet report, sending...
I20260504 13:57:28.185416 12463 ts_manager.cc:194] Registered new tserver with Master: 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331)
I20260504 13:57:28.187366 12463 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.238.194:44791
I20260504 13:57:28.188194 12310 ts_manager.cc:194] Registered new tserver with Master: 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331)
I20260504 13:57:28.332275 12874 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:28.332834 12874 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:28.334407 12874 leader_election.cc:290] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:28.335394 12318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bc3d546912c9427aaca93bdafc33dd6d" is_pre_election: true
I20260504 13:57:28.336187 12473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034" is_pre_election: true
I20260504 13:57:28.337795 12371 leader_election.cc:304] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters: 26e2451840464331a15d5291c3f09034, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:28.338569 12874 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:57:28.493917 12873 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260504 13:57:28.494433 12873 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260504 13:57:28.494988 12873 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260504 13:57:28.529114 12873 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260504 13:57:28.530019 12873 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.238.195
I20260504 13:57:28.568815 12873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.238.212:41493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.238.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.238.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.238.195
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 04 May 2026 13:43:23 UTC on e7f111948823
build id 11743
TSAN enabled
I20260504 13:57:28.570164 12873 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 13:57:28.571841 12873 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 13:57:28.585687 12881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:57:28.596060 12885 raft_consensus.cc:493] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:28.599449 12885 raft_consensus.cc:515] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:28.602955 12393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3d546912c9427aaca93bdafc33dd6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" is_pre_election: true
I20260504 13:57:28.609014 12473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3d546912c9427aaca93bdafc33dd6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034" is_pre_election: true
I20260504 13:57:28.610946 12297 leader_election.cc:304] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc3d546912c9427aaca93bdafc33dd6d; no voters: 26e2451840464331a15d5291c3f09034, 6dc832d2f7f748dbb8e3ba2f73ec98a4
I20260504 13:57:28.616343 12885 leader_election.cc:290] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:28.617075 12886 raft_consensus.cc:2749] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:57:28.691567 12863 heartbeater.cc:499] Master 127.8.238.252:35825 was elected leader, sending a full tablet report...
I20260504 13:57:28.915905 12888 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:28.921041 12888 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:28.928380 12318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bc3d546912c9427aaca93bdafc33dd6d" is_pre_election: true
I20260504 13:57:28.930197 12888 leader_election.cc:290] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:28.930277 12473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034" is_pre_election: true
I20260504 13:57:28.932169 12371 leader_election.cc:304] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters: 26e2451840464331a15d5291c3f09034, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:28.933513 12888 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:57:29.230770 12889 raft_consensus.cc:493] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:29.231539 12889 raft_consensus.cc:515] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:29.234143 12393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3d546912c9427aaca93bdafc33dd6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" is_pre_election: true
I20260504 13:57:29.237054 12473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3d546912c9427aaca93bdafc33dd6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034" is_pre_election: true
I20260504 13:57:29.240595 12297 leader_election.cc:304] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc3d546912c9427aaca93bdafc33dd6d; no voters: 26e2451840464331a15d5291c3f09034, 6dc832d2f7f748dbb8e3ba2f73ec98a4
I20260504 13:57:29.246742 12889 leader_election.cc:290] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:29.248531 12889 raft_consensus.cc:2749] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260504 13:57:29.915395 12892 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:29.915933 12892 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:29.919596 12318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bc3d546912c9427aaca93bdafc33dd6d" is_pre_election: true
I20260504 13:57:29.921852 12473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034" is_pre_election: true
I20260504 13:57:29.924031 12371 leader_election.cc:304] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters: 26e2451840464331a15d5291c3f09034, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:29.927408 12892 leader_election.cc:290] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:29.929157 12892 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:57:29.987138 12879 debug-util.cc:398] Leaking SignalData structure 0x7b0800028ca0 after lost signal to thread 12873
W20260504 13:57:30.019567 12879 kernel_stack_watchdog.cc:198] Thread 12873 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 398ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260504 13:57:28.586416 12880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:57:30.021111 12873 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.434s user 0.487s sys 0.910s
W20260504 13:57:30.021457 12873 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.435s user 0.488s sys 0.910s
W20260504 13:57:30.022684 12883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:57:30.025795 12882 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1433 milliseconds
I20260504 13:57:30.025828 12873 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260504 13:57:30.027122 12873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260504 13:57:30.029322 12873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260504 13:57:30.030666 12873 hybrid_clock.cc:648] HybridClock initialized: now 1777903050030628 us; error 41 us; skew 500 ppm
I20260504 13:57:30.031540 12873 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 13:57:30.037735 12873 webserver.cc:492] Webserver started at http://127.8.238.195:40485/ using document root <none> and password file <none>
I20260504 13:57:30.038851 12873 fs_manager.cc:362] Metadata directory not provided
I20260504 13:57:30.039080 12873 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 13:57:30.039525 12873 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260504 13:57:30.043908 12873 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/data/instance:
uuid: "7c95ca5995cc4cbba624fd3ea6367902"
format_stamp: "Formatted at 2026-05-04 13:57:30 on dist-test-slave-f9wt"
I20260504 13:57:30.044926 12873 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/wal/instance:
uuid: "7c95ca5995cc4cbba624fd3ea6367902"
format_stamp: "Formatted at 2026-05-04 13:57:30 on dist-test-slave-f9wt"
I20260504 13:57:30.052420 12873 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260504 13:57:30.059319 12898 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 13:57:30.060598 12873 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260504 13:57:30.060971 12873 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/wal
uuid: "7c95ca5995cc4cbba624fd3ea6367902"
format_stamp: "Formatted at 2026-05-04 13:57:30 on dist-test-slave-f9wt"
I20260504 13:57:30.061440 12873 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 13:57:30.124492 12873 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 13:57:30.125856 12873 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 13:57:30.126288 12873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 13:57:30.129082 12873 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260504 13:57:30.133370 12873 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260504 13:57:30.133643 12873 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260504 13:57:30.133957 12873 ts_tablet_manager.cc:616] Registered 0 tablets
I20260504 13:57:30.134102 12873 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260504 13:57:30.286267 12873 rpc_server.cc:307] RPC server started. Bound to: 127.8.238.195:36223
I20260504 13:57:30.286372 13010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.238.195:36223 every 8 connection(s)
I20260504 13:57:30.288727 12873 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/data/info.pb
I20260504 13:57:30.298180 9147 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu as pid 12873
I20260504 13:57:30.298820 9147 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/wal/instance
I20260504 13:57:30.333170 13011 heartbeater.cc:344] Connected to a master server at 127.8.238.252:35825
I20260504 13:57:30.333701 13011 heartbeater.cc:461] Registering TS with master...
I20260504 13:57:30.334977 13011 heartbeater.cc:507] Master 127.8.238.252:35825 requested a full tablet report, sending...
I20260504 13:57:30.336251 13012 heartbeater.cc:344] Connected to a master server at 127.8.238.253:35717
I20260504 13:57:30.336594 13012 heartbeater.cc:461] Registering TS with master...
I20260504 13:57:30.337388 13012 heartbeater.cc:507] Master 127.8.238.253:35717 requested a full tablet report, sending...
I20260504 13:57:30.337533 12463 ts_manager.cc:194] Registered new tserver with Master: 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223)
I20260504 13:57:30.339479 12463 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.8.238.195:40223
I20260504 13:57:30.340510 12383 ts_manager.cc:194] Registered new tserver with Master: 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223)
I20260504 13:57:30.344455 13013 heartbeater.cc:344] Connected to a master server at 127.8.238.254:38403
I20260504 13:57:30.344774 13013 heartbeater.cc:461] Registering TS with master...
I20260504 13:57:30.345397 13013 heartbeater.cc:507] Master 127.8.238.254:38403 requested a full tablet report, sending...
I20260504 13:57:30.347797 12310 ts_manager.cc:194] Registered new tserver with Master: 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223)
I20260504 13:57:30.349926 9147 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260504 13:57:30.394650 12463 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36734:
name: "default.Table_53b74b33cd8b42e883e98214b6266851"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:57:30.398072 12462 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36734:
name: "default.Table_4144f4abff494ea3b2fffc68d535e514"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:57:30.398986 12460 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36734:
name: "default.Table_741fc94d99de4e8ab34684aa3f168034"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:57:30.401225 12459 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36734:
name: "default.Table_76e3f88f42b84f1397e06f486e84d95a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:57:30.406663 12463 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_53b74b33cd8b42e883e98214b6266851 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:57:30.412431 12462 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4144f4abff494ea3b2fffc68d535e514 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:57:30.415726 12460 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_741fc94d99de4e8ab34684aa3f168034 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:57:30.417249 12459 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_76e3f88f42b84f1397e06f486e84d95a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:57:30.492986 9147 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu with pid 12420
I20260504 13:57:30.532120 13045 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:30.532719 13045 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:30.534765 13045 leader_election.cc:290] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:30.536010 12318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bc3d546912c9427aaca93bdafc33dd6d" is_pre_election: true
W20260504 13:57:30.551301 12371 connection.cc:570] client connection to 127.8.238.252:35825 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:30.551739 12371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 6 similar messages]
W20260504 13:57:30.591785 12371 leader_election.cc:336] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825): Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:30.592206 12371 leader_election.cc:304] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters: 26e2451840464331a15d5291c3f09034, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:30.592907 13045 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260504 13:57:30.528293 13024 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.238.252:35825)
W20260504 13:57:30.617846 13024 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.8.238.252:35825, user_credentials={real_user=slave}} blocked reactor thread for 91424.4us
W20260504 13:57:30.677847 12721 heartbeater.cc:646] Failed to heartbeat to 127.8.238.252:35825 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:30.685245 13050 raft_consensus.cc:493] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:30.692073 13050 raft_consensus.cc:515] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:30.695545 12393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3d546912c9427aaca93bdafc33dd6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" is_pre_election: true
I20260504 13:57:30.696095 12393 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc3d546912c9427aaca93bdafc33dd6d in term 1.
I20260504 13:57:30.697856 12296 leader_election.cc:304] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d; no voters:
W20260504 13:57:30.701830 12297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20260504 13:57:30.704898 13050 leader_election.cc:290] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:30.705382 13050 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260504 13:57:30.705722 13050 raft_consensus.cc:493] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 26e2451840464331a15d5291c3f09034)
I20260504 13:57:30.706072 13050 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 1 FOLLOWER]: Advancing to term 2
I20260504 13:57:30.724115 13050 raft_consensus.cc:515] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:30.726995 12393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc3d546912c9427aaca93bdafc33dd6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4"
W20260504 13:57:30.726961 12297 leader_election.cc:336] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825): Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:30.727242 13050 leader_election.cc:290] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 election: Requested vote from peers 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:30.727560 12393 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 1 FOLLOWER]: Advancing to term 2
W20260504 13:57:30.731680 12297 leader_election.cc:336] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825): Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:30.735893 12393 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc3d546912c9427aaca93bdafc33dd6d in term 2.
I20260504 13:57:30.748009 12296 leader_election.cc:304] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d; no voters: 26e2451840464331a15d5291c3f09034
I20260504 13:57:30.749092 13050 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 2 FOLLOWER]: Leader election won for term 2
I20260504 13:57:30.749780 13050 raft_consensus.cc:697] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [term 2 LEADER]: Becoming Leader. State: Replica: bc3d546912c9427aaca93bdafc33dd6d, State: Running, Role: LEADER
W20260504 13:57:30.751868 12863 heartbeater.cc:646] Failed to heartbeat to 127.8.238.252:35825 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:30.750730 13050 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:30.762506 13052 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc3d546912c9427aaca93bdafc33dd6d. Latest consensus state: current_term: 2 leader_uuid: "bc3d546912c9427aaca93bdafc33dd6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:30.765930 13052 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [sys.catalog]: This master's current role is: LEADER
I20260504 13:57:30.786495 13056 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260504 13:57:30.802874 13056 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260504 13:57:30.806713 13056 catalog_manager.cc:1269] Loaded cluster ID: 8a9361cd032446ef9af554d6cfd5f3e8
I20260504 13:57:30.806991 13056 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260504 13:57:30.811568 13056 catalog_manager.cc:1514] Loading token signing keys...
I20260504 13:57:30.815285 13056 catalog_manager.cc:6055] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d: Loaded TSK: 0
I20260504 13:57:30.816546 13056 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260504 13:57:30.822634 13056 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260504 13:57:30.827157 13056 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260504 13:57:30.843145 13011 heartbeater.cc:499] Master 127.8.238.252:35825 was elected leader, sending a full tablet report...
W20260504 13:57:30.864269 13011 heartbeater.cc:646] Failed to heartbeat to 127.8.238.252:35825 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:30.847838 12308 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44828:
name: "default.Table_4144f4abff494ea3b2fffc68d535e514"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:57:30.848721 12307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44828:
name: "default.Table_53b74b33cd8b42e883e98214b6266851"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:57:30.850152 12310 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44828:
name: "default.Table_76e3f88f42b84f1397e06f486e84d95a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260504 13:57:30.898101 12310 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_76e3f88f42b84f1397e06f486e84d95a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:57:30.900522 12307 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_53b74b33cd8b42e883e98214b6266851 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260504 13:57:30.901840 12308 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4144f4abff494ea3b2fffc68d535e514 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:57:30.843392 12309 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44828:
name: "default.Table_741fc94d99de4e8ab34684aa3f168034"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:57:31.118705 12310 catalog_manager.cc:2174] created HMS catalog entry for table [id=8b7a3413a9f44a8c915ffa9ff2484c41]
W20260504 13:57:31.135067 12309 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_741fc94d99de4e8ab34684aa3f168034 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260504 13:57:31.143137 12393 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Refusing update from remote peer bc3d546912c9427aaca93bdafc33dd6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20260504 13:57:31.144718 12297 consensus_peers.cc:597] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d -> Peer 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825): Couldn't send request to peer 26e2451840464331a15d5291c3f09034. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:57:31.144807 13050 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260504 13:57:31.181428 13059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc3d546912c9427aaca93bdafc33dd6d. Latest consensus state: current_term: 2 leader_uuid: "bc3d546912c9427aaca93bdafc33dd6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:31.182166 13059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:57:31.188299 13050 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "bc3d546912c9427aaca93bdafc33dd6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:31.188959 13050 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d [sys.catalog]: This master's current role is: LEADER
I20260504 13:57:31.194206 13059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "bc3d546912c9427aaca93bdafc33dd6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:31.194407 12310 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44828:
table { table_name: "default.Table_76e3f88f42b84f1397e06f486e84d95a" } new_table_name: "default.Table_14adc72f313248df95e58d7dfaa02df9" modify_external_catalogs: true
I20260504 13:57:31.194836 13059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:57:31.194765 12308 catalog_manager.cc:2174] created HMS catalog entry for table [id=be3e1aa95f514f9db45c7887bc578614]
I20260504 13:57:31.210270 12308 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44828:
table { table_name: "default.Table_4144f4abff494ea3b2fffc68d535e514" } new_table_name: "default.Table_c4220efa2c914d1391eb4d7f5459d3c3" modify_external_catalogs: true
I20260504 13:57:31.282440 12946 tablet_service.cc:1511] Processing CreateTablet for tablet 87d2ae9451964160836b7d6aa6d95587 (DEFAULT_TABLE table=default.table_76e3f88f42b84f1397e06f486e84d95a [id=8b7a3413a9f44a8c915ffa9ff2484c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:31.291538 12945 tablet_service.cc:1511] Processing CreateTablet for tablet a9cfb4321395442891167633a8c76dea (DEFAULT_TABLE table=default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:31.293578 12945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cfb4321395442891167633a8c76dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:31.297425 12946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87d2ae9451964160836b7d6aa6d95587. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:31.306106 12655 tablet_service.cc:1511] Processing CreateTablet for tablet 87d2ae9451964160836b7d6aa6d95587 (DEFAULT_TABLE table=default.table_76e3f88f42b84f1397e06f486e84d95a [id=8b7a3413a9f44a8c915ffa9ff2484c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:31.308058 12655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87d2ae9451964160836b7d6aa6d95587. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:31.313608 12654 tablet_service.cc:1511] Processing CreateTablet for tablet a9cfb4321395442891167633a8c76dea (DEFAULT_TABLE table=default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:31.314785 12654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cfb4321395442891167633a8c76dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:31.352722 12798 tablet_service.cc:1511] Processing CreateTablet for tablet 87d2ae9451964160836b7d6aa6d95587 (DEFAULT_TABLE table=default.table_76e3f88f42b84f1397e06f486e84d95a [id=8b7a3413a9f44a8c915ffa9ff2484c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:31.354619 12798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87d2ae9451964160836b7d6aa6d95587. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:31.355502 12797 tablet_service.cc:1511] Processing CreateTablet for tablet a9cfb4321395442891167633a8c76dea (DEFAULT_TABLE table=default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:31.359469 13072 tablet_bootstrap.cc:492] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6: Bootstrap starting.
I20260504 13:57:31.359870 12797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9cfb4321395442891167633a8c76dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:31.363471 13013 heartbeater.cc:499] Master 127.8.238.254:38403 was elected leader, sending a full tablet report...
I20260504 13:57:31.371990 12305 master_service.cc:954] ReplaceTablet: received request to replace tablet a9cfb4321395442891167633a8c76dea from {username='slave'} at 127.0.0.1:44828
I20260504 13:57:31.373028 12304 master_service.cc:954] ReplaceTablet: received request to replace tablet a9cfb4321395442891167633a8c76dea from {username='slave'} at 127.0.0.1:44828
I20260504 13:57:31.390166 13071 tablet_bootstrap.cc:492] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902: Bootstrap starting.
I20260504 13:57:31.391953 13072 tablet_bootstrap.cc:654] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:31.403837 12306 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44828:
name: "default.Table_91bfcb8f6cb843cea4deee25d923b07c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:57:31.408680 12305 catalog_manager.cc:5958] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d: Sending DeleteTablet for 3 replicas of tablet a9cfb4321395442891167633a8c76dea
I20260504 13:57:31.410719 12655 tablet_service.cc:1558] Processing DeleteTablet for tablet a9cfb4321395442891167633a8c76dea with delete_type TABLET_DATA_DELETED (replaced by tablet 4c771aa2ad854224b235afb92cbb5ea5) from {username='slave'} at 127.0.0.1:47760
I20260504 13:57:31.411206 12796 tablet_service.cc:1558] Processing DeleteTablet for tablet a9cfb4321395442891167633a8c76dea with delete_type TABLET_DATA_DELETED (replaced by tablet 4c771aa2ad854224b235afb92cbb5ea5) from {username='slave'} at 127.0.0.1:49474
I20260504 13:57:31.411298 12304 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44828:
name: "default.Table_2dc3318442a1464eb3c4db4f57a86c83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:57:31.420482 13071 tablet_bootstrap.cc:654] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:31.423858 13071 log.cc:826] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902: Log is configured to *not* fsync() on all Append() calls
W20260504 13:57:31.428016 12295 catalog_manager.cc:4982] TS 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229): delete failed for tablet a9cfb4321395442891167633a8c76dea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9cfb4321395442891167633a8c76dea already in progress: creating tablet
I20260504 13:57:31.440379 13076 tablet_bootstrap.cc:492] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3: Bootstrap starting.
I20260504 13:57:31.441529 12305 catalog_manager.cc:6741] ReplaceTablet: tablet a9cfb4321395442891167633a8c76dea deleted and replaced by tablet 4c771aa2ad854224b235afb92cbb5ea5
I20260504 13:57:31.445967 12655 tablet_service.cc:1511] Processing CreateTablet for tablet 4c771aa2ad854224b235afb92cbb5ea5 (DEFAULT_TABLE table=default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:31.447228 12655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c771aa2ad854224b235afb92cbb5ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:31.445212 13071 tablet_bootstrap.cc:492] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902: No bootstrap required, opened a new log
W20260504 13:57:31.455971 12296 catalog_manager.cc:4982] TS 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331): delete failed for tablet a9cfb4321395442891167633a8c76dea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9cfb4321395442891167633a8c76dea already in progress: creating tablet
I20260504 13:57:31.456804 13071 ts_tablet_manager.cc:1403] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.007s
I20260504 13:57:31.461146 12946 tablet_service.cc:1558] Processing DeleteTablet for tablet a9cfb4321395442891167633a8c76dea with delete_type TABLET_DATA_DELETED (replaced by tablet 4c771aa2ad854224b235afb92cbb5ea5) from {username='slave'} at 127.0.0.1:44786
I20260504 13:57:31.463606 12945 tablet_service.cc:1511] Processing CreateTablet for tablet 4c771aa2ad854224b235afb92cbb5ea5 (DEFAULT_TABLE table=default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:31.464704 12945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c771aa2ad854224b235afb92cbb5ea5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260504 13:57:31.466760 12296 catalog_manager.cc:4982] TS 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223): delete failed for tablet a9cfb4321395442891167633a8c76dea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9cfb4321395442891167633a8c76dea already in progress: creating tablet
I20260504 13:57:31.472422 13072 log.cc:826] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6: Log is configured to *not* fsync() on all Append() calls
I20260504 13:57:31.456185 12797 tablet_service.cc:1511] Processing CreateTablet for tablet 4c771aa2ad854224b235afb92cbb5ea5 (DEFAULT_TABLE table=default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:31.480190 12797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c771aa2ad854224b235afb92cbb5ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:31.490186 13072 tablet_bootstrap.cc:492] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6: No bootstrap required, opened a new log
I20260504 13:57:31.490832 13072 ts_tablet_manager.cc:1403] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6: Time spent bootstrapping tablet: real 0.132s user 0.012s sys 0.004s
13:57:31.494 [HMS - ERROR - pool-6-thread-3] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_53b74b33cd8b42e883e98214b6266851 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at jdk.proxy2/jdk.proxy2.$Proxy27.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
W20260504 13:57:31.497123 12307 catalog_manager.cc:2170] Already present: failed to create HMS catalog entry for table [id=d53d0b77bda242f9a55c1aaea9eb23cc]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_53b74b33cd8b42e883e98214b6266851 already exists)
I20260504 13:57:31.501438 12307 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44828:
name: "default.Table_398fe552fc9a43ac8875babd6d5001fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:57:31.508709 13076 tablet_bootstrap.cc:654] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:31.509905 9147 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
/tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.252:35825
--webserver_interface=127.8.238.252
--webserver_port=35067
--builtin_ntp_servers=127.8.238.212:41493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--hive_metastore_uris=thrift://127.0.0.1:39899
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260504 13:57:31.517164 13072 raft_consensus.cc:359] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:31.518215 13072 raft_consensus.cc:385] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:31.518577 13072 raft_consensus.cc:740] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910fbf5d51f843609f5caf25e52beae6, State: Initialized, Role: FOLLOWER
I20260504 13:57:31.519505 13072 consensus_queue.cc:260] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:31.527016 13076 log.cc:826] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3: Log is configured to *not* fsync() on all Append() calls
I20260504 13:57:31.530190 13072 ts_tablet_manager.cc:1434] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6: Time spent starting tablet: real 0.039s user 0.028s sys 0.008s
I20260504 13:57:31.539913 12723 heartbeater.cc:499] Master 127.8.238.254:38403 was elected leader, sending a full tablet report...
I20260504 13:57:31.540246 13072 tablet_bootstrap.cc:492] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6: Bootstrap starting.
I20260504 13:57:31.546957 13072 tablet_bootstrap.cc:654] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:31.551455 12655 tablet_service.cc:1558] Processing DeleteTablet for tablet a9cfb4321395442891167633a8c76dea with delete_type TABLET_DATA_DELETED (replaced by tablet 4c771aa2ad854224b235afb92cbb5ea5) from {username='slave'} at 127.0.0.1:47760
I20260504 13:57:31.552700 13074 tablet_replica.cc:333] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6: stopping tablet replica
I20260504 13:57:31.553524 13074 raft_consensus.cc:2243] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:57:31.554059 13074 raft_consensus.cc:2272] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:57:31.555388 13076 tablet_bootstrap.cc:492] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3: No bootstrap required, opened a new log
I20260504 13:57:31.555912 13076 ts_tablet_manager.cc:1403] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent bootstrapping tablet: real 0.116s user 0.015s sys 0.003s
I20260504 13:57:31.556870 13074 ts_tablet_manager.cc:1916] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:57:31.582206 13071 raft_consensus.cc:359] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } }
I20260504 13:57:31.583243 13072 tablet_bootstrap.cc:492] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6: No bootstrap required, opened a new log
I20260504 13:57:31.583388 13071 raft_consensus.cc:385] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:31.583854 13072 ts_tablet_manager.cc:1403] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.006s
I20260504 13:57:31.583745 13071 raft_consensus.cc:740] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c95ca5995cc4cbba624fd3ea6367902, State: Initialized, Role: FOLLOWER
I20260504 13:57:31.586800 13072 raft_consensus.cc:359] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } }
I20260504 13:57:31.587589 13072 raft_consensus.cc:385] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:31.587935 13072 raft_consensus.cc:740] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910fbf5d51f843609f5caf25e52beae6, State: Initialized, Role: FOLLOWER
I20260504 13:57:31.587631 13071 consensus_queue.cc:260] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } }
I20260504 13:57:31.588636 13072 consensus_queue.cc:260] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } }
I20260504 13:57:31.589615 13074 ts_tablet_manager.cc:1929] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:57:31.592249 13074 log.cc:1199] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/a9cfb4321395442891167633a8c76dea
I20260504 13:57:31.594295 13074 ts_tablet_manager.cc:1950] T a9cfb4321395442891167633a8c76dea P 910fbf5d51f843609f5caf25e52beae6: Deleting consensus metadata
I20260504 13:57:31.593308 13072 ts_tablet_manager.cc:1434] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260504 13:57:31.597676 12295 catalog_manager.cc:5002] TS 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229): tablet a9cfb4321395442891167633a8c76dea (table default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614]) successfully deleted
I20260504 13:57:31.603730 13072 tablet_bootstrap.cc:492] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6: Bootstrap starting.
I20260504 13:57:31.607447 13071 ts_tablet_manager.cc:1434] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902: Time spent starting tablet: real 0.150s user 0.021s sys 0.015s
I20260504 13:57:31.609464 13071 tablet_bootstrap.cc:492] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902: Bootstrap starting.
I20260504 13:57:31.618420 13076 raft_consensus.cc:359] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:31.620937 13072 tablet_bootstrap.cc:654] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:31.620992 12302 master_service.cc:954] ReplaceTablet: received request to replace tablet 87d2ae9451964160836b7d6aa6d95587 from {username='slave'} at 127.0.0.1:44828
I20260504 13:57:31.622635 13071 tablet_bootstrap.cc:654] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:31.650673 12305 master_service.cc:954] ReplaceTablet: received request to replace tablet 4c771aa2ad854224b235afb92cbb5ea5 from {username='slave'} at 127.0.0.1:44828
I20260504 13:57:31.654974 12302 catalog_manager.cc:5958] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d: Sending DeleteTablet for 3 replicas of tablet 87d2ae9451964160836b7d6aa6d95587
I20260504 13:57:31.656836 12302 catalog_manager.cc:6741] ReplaceTablet: tablet 87d2ae9451964160836b7d6aa6d95587 deleted and replaced by tablet 6f6388ce6c7f49db91576194fe741a73
I20260504 13:57:31.659430 12655 tablet_service.cc:1558] Processing DeleteTablet for tablet 87d2ae9451964160836b7d6aa6d95587 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f6388ce6c7f49db91576194fe741a73) from {username='slave'} at 127.0.0.1:47760
I20260504 13:57:31.660269 13074 tablet_replica.cc:333] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6: stopping tablet replica
I20260504 13:57:31.661105 13074 raft_consensus.cc:2243] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:57:31.661589 13074 raft_consensus.cc:2272] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:57:31.662992 12945 tablet_service.cc:1558] Processing DeleteTablet for tablet 87d2ae9451964160836b7d6aa6d95587 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f6388ce6c7f49db91576194fe741a73) from {username='slave'} at 127.0.0.1:44786
I20260504 13:57:31.663867 13078 tablet_replica.cc:333] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902: stopping tablet replica
I20260504 13:57:31.664660 13078 raft_consensus.cc:2243] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:57:31.665195 13078 raft_consensus.cc:2272] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:57:31.669116 13078 ts_tablet_manager.cc:1916] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:57:31.669790 13074 ts_tablet_manager.cc:1916] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:57:31.671802 12309 catalog_manager.cc:2174] created HMS catalog entry for table [id=ab5674daf2194017b802477a5586ec14]
F20260504 13:57:31.687665 12575 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1777903050, eventType=CREATE_TABLE, dbName=default, tableName=table_76e3f88f42b84f1397e06f486e84d95a, message={"server":"","servicePrincipal":"","db":"default","table":"table_76e3f88f42b84f1397e06f486e84d95a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_76e3f88f42b84f1397e06f486e84d95a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777903050},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/warehouse/table_76e3f88f42b84f1397e06f486e84d95a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_76e3f88f42b84f1397e06f486e84d95a\",\"kudu.table_id\":\"8b7a3413a9f44a8c915ffa9ff2484c41\",\"transient_lastDdlTime\":\"1777903050\",\"comment\":\"\",\"kudu.master_addresses\":\"127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8a9361cd032446ef9af554d6cfd5f3e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777903050,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777903050, eventType=CREATE_TABLE, dbName=default, tableName=table_53b74b33cd8b42e883e98214b6266851, message={"server":"","servicePrincipal":"","db":"default","table":"table_53b74b33cd8b42e883e98214b6266851","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_53b74b33cd8b42e883e98214b6266851\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777903050},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/warehouse/table_53b74b33cd8b42e883e98214b6266851\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_53b74b33cd8b42e883e98214b6266851\",\"kudu.table_id\":\"389bb89032254cbd88075a6da721baf6\",\"transient_lastDdlTime\":\"1777903050\",\"comment\":\"\",\"kudu.master_addresses\":\"127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8a9361cd032446ef9af554d6cfd5f3e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777903050,"files":[]}, messageFormat=json-0.2, catName=hive);
I20260504 13:57:31.662365 12797 tablet_service.cc:1558] Processing DeleteTablet for tablet 87d2ae9451964160836b7d6aa6d95587 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f6388ce6c7f49db91576194fe741a73) from {username='slave'} at 127.0.0.1:49474
I20260504 13:57:31.691354 13076 raft_consensus.cc:385] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:31.691809 13076 raft_consensus.cc:740] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3517f3498e5e4e9cb5cbfa239e4999e3, State: Initialized, Role: FOLLOWER
I20260504 13:57:31.694913 13071 tablet_bootstrap.cc:492] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902: No bootstrap required, opened a new log
I20260504 13:57:31.695382 13071 ts_tablet_manager.cc:1403] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902: Time spent bootstrapping tablet: real 0.086s user 0.011s sys 0.000s
*** Check failure stack trace: ***
W20260504 13:57:31.694216 12296 catalog_manager.cc:4982] TS 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331): delete failed for tablet 87d2ae9451964160836b7d6aa6d95587 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87d2ae9451964160836b7d6aa6d95587 already in progress: creating tablet
I20260504 13:57:31.697455 13076 consensus_queue.cc:260] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:31.709940 13071 raft_consensus.cc:359] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:31.710703 13071 raft_consensus.cc:385] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:31.721139 13074 ts_tablet_manager.cc:1929] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:57:31.721540 13074 log.cc:1199] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/87d2ae9451964160836b7d6aa6d95587
I20260504 13:57:31.722185 13074 ts_tablet_manager.cc:1950] T 87d2ae9451964160836b7d6aa6d95587 P 910fbf5d51f843609f5caf25e52beae6: Deleting consensus metadata
I20260504 13:57:31.710988 13071 raft_consensus.cc:740] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c95ca5995cc4cbba624fd3ea6367902, State: Initialized, Role: FOLLOWER
I20260504 13:57:31.724884 12305 catalog_manager.cc:5958] T 00000000000000000000000000000000 P bc3d546912c9427aaca93bdafc33dd6d: Sending DeleteTablet for 3 replicas of tablet 4c771aa2ad854224b235afb92cbb5ea5
I20260504 13:57:31.724515 13071 consensus_queue.cc:260] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:31.726724 12655 tablet_service.cc:1558] Processing DeleteTablet for tablet 4c771aa2ad854224b235afb92cbb5ea5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8896d8df10394e30ac15f3b04625b13b) from {username='slave'} at 127.0.0.1:47760
I20260504 13:57:31.728600 12305 catalog_manager.cc:6741] ReplaceTablet: tablet 4c771aa2ad854224b235afb92cbb5ea5 deleted and replaced by tablet 8896d8df10394e30ac15f3b04625b13b
I20260504 13:57:31.728940 13072 tablet_bootstrap.cc:492] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6: No bootstrap required, opened a new log
I20260504 13:57:31.729398 13072 ts_tablet_manager.cc:1403] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6: Time spent bootstrapping tablet: real 0.126s user 0.026s sys 0.026s
I20260504 13:57:31.729837 12797 tablet_service.cc:1558] Processing DeleteTablet for tablet 4c771aa2ad854224b235afb92cbb5ea5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8896d8df10394e30ac15f3b04625b13b) from {username='slave'} at 127.0.0.1:49474
I20260504 13:57:31.732308 12945 tablet_service.cc:1558] Processing DeleteTablet for tablet 4c771aa2ad854224b235afb92cbb5ea5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8896d8df10394e30ac15f3b04625b13b) from {username='slave'} at 127.0.0.1:44786
W20260504 13:57:31.733103 12296 catalog_manager.cc:4982] TS 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331): delete failed for tablet 4c771aa2ad854224b235afb92cbb5ea5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c771aa2ad854224b235afb92cbb5ea5 already in progress: creating tablet
I20260504 13:57:31.745136 12945 tablet_service.cc:1558] Processing DeleteTablet for tablet a9cfb4321395442891167633a8c76dea with delete_type TABLET_DATA_DELETED (replaced by tablet 4c771aa2ad854224b235afb92cbb5ea5) from {username='slave'} at 127.0.0.1:44786
I20260504 13:57:31.748760 12865 heartbeater.cc:499] Master 127.8.238.254:38403 was elected leader, sending a full tablet report...
I20260504 13:57:31.754362 13078 ts_tablet_manager.cc:1929] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:57:31.757229 13078 log.cc:1199] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/87d2ae9451964160836b7d6aa6d95587
I20260504 13:57:31.760557 13088 raft_consensus.cc:493] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:31.761169 13088 raft_consensus.cc:515] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:31.765965 13088 leader_election.cc:290] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229), 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223)
@ 0x7f041fd72950 google::LogMessage::Flush() at ??:0
I20260504 13:57:31.777354 13076 ts_tablet_manager.cc:1434] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent starting tablet: real 0.221s user 0.038s sys 0.001s
I20260504 13:57:31.758455 13071 ts_tablet_manager.cc:1434] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902: Time spent starting tablet: real 0.063s user 0.007s sys 0.000s
I20260504 13:57:31.782433 13071 tablet_bootstrap.cc:492] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902: Bootstrap starting.
I20260504 13:57:31.786973 12966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9cfb4321395442891167633a8c76dea" candidate_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c95ca5995cc4cbba624fd3ea6367902" is_pre_election: true
I20260504 13:57:31.787807 12966 raft_consensus.cc:2468] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3517f3498e5e4e9cb5cbfa239e4999e3 in term 0.
I20260504 13:57:31.789234 13078 ts_tablet_manager.cc:1950] T 87d2ae9451964160836b7d6aa6d95587 P 7c95ca5995cc4cbba624fd3ea6367902: Deleting consensus metadata
I20260504 13:57:31.789533 12753 leader_election.cc:304] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3517f3498e5e4e9cb5cbfa239e4999e3, 7c95ca5995cc4cbba624fd3ea6367902; no voters:
I20260504 13:57:31.790424 13088 raft_consensus.cc:2804] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:57:31.790786 13088 raft_consensus.cc:493] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:57:31.791102 13088 raft_consensus.cc:3060] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:57:31.791035 12295 catalog_manager.cc:5002] TS 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229): tablet 87d2ae9451964160836b7d6aa6d95587 (table default.table_76e3f88f42b84f1397e06f486e84d95a [id=8b7a3413a9f44a8c915ffa9ff2484c41]) successfully deleted
W20260504 13:57:31.791808 12295 catalog_manager.cc:4982] TS 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229): delete failed for tablet 4c771aa2ad854224b235afb92cbb5ea5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c771aa2ad854224b235afb92cbb5ea5 already in progress: creating tablet
I20260504 13:57:31.795748 13072 raft_consensus.cc:359] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:31.796418 13072 raft_consensus.cc:385] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:31.796686 13072 raft_consensus.cc:740] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910fbf5d51f843609f5caf25e52beae6, State: Initialized, Role: FOLLOWER
I20260504 13:57:31.797515 13072 consensus_queue.cc:260] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:31.799927 13072 ts_tablet_manager.cc:1434] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6: Time spent starting tablet: real 0.070s user 0.003s sys 0.004s
I20260504 13:57:31.801582 13076 tablet_bootstrap.cc:492] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3: Bootstrap starting.
@ 0x7f041fd726bf google::LogMessage::~LogMessage() at ??:0
I20260504 13:57:31.805384 13078 tablet_replica.cc:333] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902: stopping tablet replica
I20260504 13:57:31.805706 12296 catalog_manager.cc:5002] TS 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223): tablet 87d2ae9451964160836b7d6aa6d95587 (table default.table_76e3f88f42b84f1397e06f486e84d95a [id=8b7a3413a9f44a8c915ffa9ff2484c41]) successfully deleted
I20260504 13:57:31.806166 13078 raft_consensus.cc:2243] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260504 13:57:31.806414 12296 catalog_manager.cc:4982] TS 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223): delete failed for tablet 4c771aa2ad854224b235afb92cbb5ea5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c771aa2ad854224b235afb92cbb5ea5 already in progress: creating tablet
I20260504 13:57:31.806648 13078 raft_consensus.cc:2272] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:57:31.803505 13088 raft_consensus.cc:515] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:31.809309 13088 leader_election.cc:290] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [CANDIDATE]: Term 1 election: Requested vote from peers 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229), 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223)
I20260504 13:57:31.811340 12966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9cfb4321395442891167633a8c76dea" candidate_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c95ca5995cc4cbba624fd3ea6367902"
W20260504 13:57:31.812729 12753 leader_election.cc:343] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223): Illegal state: must be running to vote when last-logged opid is not known
I20260504 13:57:31.820523 13071 tablet_bootstrap.cc:654] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:31.824574 13078 ts_tablet_manager.cc:1916] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:57:31.830475 12655 tablet_service.cc:1511] Processing CreateTablet for tablet 6f6388ce6c7f49db91576194fe741a73 (DEFAULT_TABLE table=default.table_76e3f88f42b84f1397e06f486e84d95a [id=8b7a3413a9f44a8c915ffa9ff2484c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:31.831560 12655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f6388ce6c7f49db91576194fe741a73. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f042ae322bc kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
W20260504 13:57:31.834901 12725 tablet.cc:2404] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260504 13:57:31.836721 13076 tablet_bootstrap.cc:654] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3: Neither blocks nor log segments found. Creating new log.
W20260504 13:57:31.839983 12752 leader_election.cc:343] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229): Not found: Tablet not found: a9cfb4321395442891167633a8c76dea
W20260504 13:57:31.841286 12752 leader_election.cc:343] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229): Not found: Tablet not found: a9cfb4321395442891167633a8c76dea
I20260504 13:57:31.841666 12752 leader_election.cc:304] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3517f3498e5e4e9cb5cbfa239e4999e3; no voters: 7c95ca5995cc4cbba624fd3ea6367902, 910fbf5d51f843609f5caf25e52beae6
I20260504 13:57:31.842512 13088 raft_consensus.cc:2749] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260504 13:57:31.845083 13071 tablet_bootstrap.cc:492] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902: No bootstrap required, opened a new log
I20260504 13:57:31.845515 13071 ts_tablet_manager.cc:1403] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.006s
I20260504 13:57:31.845917 12309 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44828:
table { table_name: "default.Table_741fc94d99de4e8ab34684aa3f168034" } modify_external_catalogs: true
I20260504 13:57:31.846626 12309 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44828:
table { table_name: "default.Table_741fc94d99de4e8ab34684aa3f168034" } modify_external_catalogs: true
I20260504 13:57:31.853076 12945 tablet_service.cc:1511] Processing CreateTablet for tablet 6f6388ce6c7f49db91576194fe741a73 (DEFAULT_TABLE table=default.table_76e3f88f42b84f1397e06f486e84d95a [id=8b7a3413a9f44a8c915ffa9ff2484c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:31.854143 12945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f6388ce6c7f49db91576194fe741a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:31.856256 13071 raft_consensus.cc:359] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:31.856931 13071 raft_consensus.cc:385] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:31.857192 13071 raft_consensus.cc:740] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c95ca5995cc4cbba624fd3ea6367902, State: Initialized, Role: FOLLOWER
I20260504 13:57:31.860056 13076 tablet_bootstrap.cc:492] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3: No bootstrap required, opened a new log
I20260504 13:57:31.860471 13076 ts_tablet_manager.cc:1403] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.000s
I20260504 13:57:31.860342 12797 tablet_service.cc:1511] Processing CreateTablet for tablet 6f6388ce6c7f49db91576194fe741a73 (DEFAULT_TABLE table=default.table_76e3f88f42b84f1397e06f486e84d95a [id=8b7a3413a9f44a8c915ffa9ff2484c41]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f042ae31a7c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260504 13:57:31.861443 12797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f6388ce6c7f49db91576194fe741a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:31.863287 13076 raft_consensus.cc:359] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } }
I20260504 13:57:31.863948 13076 raft_consensus.cc:385] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:31.864571 13071 consensus_queue.cc:260] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:31.871420 13071 ts_tablet_manager.cc:1434] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902: Time spent starting tablet: real 0.026s user 0.007s sys 0.001s
I20260504 13:57:31.864213 13076 raft_consensus.cc:740] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3517f3498e5e4e9cb5cbfa239e4999e3, State: Initialized, Role: FOLLOWER
I20260504 13:57:31.872730 13076 consensus_queue.cc:260] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } }
I20260504 13:57:31.874904 13076 ts_tablet_manager.cc:1434] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260504 13:57:31.883569 13076 tablet_bootstrap.cc:492] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3: Bootstrap starting.
@ 0x7f042ae35af2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260504 13:57:31.889564 13076 tablet_bootstrap.cc:654] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:31.880448 13078 ts_tablet_manager.cc:1929] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:57:31.891990 13078 log.cc:1199] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/a9cfb4321395442891167633a8c76dea
I20260504 13:57:31.892700 13078 ts_tablet_manager.cc:1950] T a9cfb4321395442891167633a8c76dea P 7c95ca5995cc4cbba624fd3ea6367902: Deleting consensus metadata
I20260504 13:57:31.905138 13076 tablet_bootstrap.cc:492] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3: No bootstrap required, opened a new log
I20260504 13:57:31.905608 13076 ts_tablet_manager.cc:1403] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.014s
@ 0x7f042ae35aaa _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260504 13:57:31.908708 13076 raft_consensus.cc:359] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:31.909564 13076 raft_consensus.cc:385] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:31.909893 13076 raft_consensus.cc:740] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3517f3498e5e4e9cb5cbfa239e4999e3, State: Initialized, Role: FOLLOWER
I20260504 13:57:31.910822 13076 consensus_queue.cc:260] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
W20260504 13:57:31.914177 12870 tablet.cc:2404] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260504 13:57:31.918128 13076 ts_tablet_manager.cc:1434] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260504 13:57:31.922745 12296 catalog_manager.cc:5002] TS 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223): tablet a9cfb4321395442891167633a8c76dea (table default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614]) successfully deleted
@ 0x7f042ae35a3a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260504 13:57:31.931545 12654 tablet_service.cc:1558] Processing DeleteTablet for tablet 4c771aa2ad854224b235afb92cbb5ea5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8896d8df10394e30ac15f3b04625b13b) from {username='slave'} at 127.0.0.1:47760
I20260504 13:57:31.932293 13074 tablet_replica.cc:333] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6: stopping tablet replica
I20260504 13:57:31.933075 13074 raft_consensus.cc:2243] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:57:31.933571 13074 raft_consensus.cc:2272] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:57:31.940640 13074 ts_tablet_manager.cc:1916] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:57:31.946583 13071 tablet_bootstrap.cc:492] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902: Bootstrap starting.
I20260504 13:57:31.949509 12946 tablet_service.cc:1558] Processing DeleteTablet for tablet 4c771aa2ad854224b235afb92cbb5ea5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8896d8df10394e30ac15f3b04625b13b) from {username='slave'} at 127.0.0.1:44786
I20260504 13:57:31.950300 13078 tablet_replica.cc:333] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902: stopping tablet replica
I20260504 13:57:31.950966 13078 raft_consensus.cc:2243] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Raft consensus shutting down.
@ 0x7f042ae35a02 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260504 13:57:31.951457 13078 raft_consensus.cc:2272] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:57:31.953001 13071 tablet_bootstrap.cc:654] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:31.953898 13078 ts_tablet_manager.cc:1916] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:57:31.955924 12796 tablet_service.cc:1558] Processing DeleteTablet for tablet a9cfb4321395442891167633a8c76dea with delete_type TABLET_DATA_DELETED (replaced by tablet 4c771aa2ad854224b235afb92cbb5ea5) from {username='slave'} at 127.0.0.1:49474
I20260504 13:57:31.956794 13075 tablet_replica.cc:333] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3: stopping tablet replica
I20260504 13:57:31.956861 13076 tablet_bootstrap.cc:492] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3: Bootstrap starting.
I20260504 13:57:31.957669 13075 raft_consensus.cc:2243] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:57:31.958216 13075 raft_consensus.cc:2272] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:57:31.959947 12796 tablet_service.cc:1558] Processing DeleteTablet for tablet 87d2ae9451964160836b7d6aa6d95587 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f6388ce6c7f49db91576194fe741a73) from {username='slave'} at 127.0.0.1:49474
I20260504 13:57:31.962730 13075 ts_tablet_manager.cc:1916] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:57:31.963313 13076 tablet_bootstrap.cc:654] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:31.968593 13071 tablet_bootstrap.cc:492] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902: No bootstrap required, opened a new log
I20260504 13:57:31.968982 13071 ts_tablet_manager.cc:1403] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.000s
I20260504 13:57:31.970145 12796 tablet_service.cc:1558] Processing DeleteTablet for tablet 4c771aa2ad854224b235afb92cbb5ea5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8896d8df10394e30ac15f3b04625b13b) from {username='slave'} at 127.0.0.1:49474
I20260504 13:57:31.971616 13071 raft_consensus.cc:359] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:31.972287 13071 raft_consensus.cc:385] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:31.972559 13071 raft_consensus.cc:740] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c95ca5995cc4cbba624fd3ea6367902, State: Initialized, Role: FOLLOWER
I20260504 13:57:31.973215 13071 consensus_queue.cc:260] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:31.975078 13071 ts_tablet_manager.cc:1434] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260504 13:57:31.981297 12303 master_service.cc:954] ReplaceTablet: received request to replace tablet 4c771aa2ad854224b235afb92cbb5ea5 from {username='slave'} at 127.0.0.1:44828
@ 0x7f042ae34cfe std::__1::__function::__func<>::operator()() at ??:0
@ 0x5b3bc5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5b39fa std::__1::function<>::operator()() at ??:?
I20260504 13:57:31.988176 12303 master_service.cc:954] ReplaceTablet: received request to replace tablet 4c771aa2ad854224b235afb92cbb5ea5 from {username='slave'} at 127.0.0.1:44828
@ 0x7f0421f98fbb kudu::Thread::SuperviseThread() at ??:0
@ 0x479510 __tsan_thread_start_func at ??:?
@ 0x7f04236946db start_thread at ??:0
I20260504 13:57:32.026978 12303 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44828:
name: "default.Table_8366b20cb82349ea8ee8c637ca836adf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:57:32.032691 13072 tablet_bootstrap.cc:492] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6: Bootstrap starting.
@ 0x7f041ea1761f clone at ??:0
I20260504 13:57:32.039755 13072 tablet_bootstrap.cc:654] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:32.044690 13074 ts_tablet_manager.cc:1929] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:57:32.045058 13074 log.cc:1199] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/4c771aa2ad854224b235afb92cbb5ea5
I20260504 13:57:32.045199 13076 tablet_bootstrap.cc:492] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3: No bootstrap required, opened a new log
I20260504 13:57:32.045614 13076 ts_tablet_manager.cc:1403] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.050s
I20260504 13:57:32.045666 13074 ts_tablet_manager.cc:1950] T 4c771aa2ad854224b235afb92cbb5ea5 P 910fbf5d51f843609f5caf25e52beae6: Deleting consensus metadata
I20260504 13:57:32.048712 13076 raft_consensus.cc:359] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:32.049338 13076 raft_consensus.cc:385] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:32.049613 13076 raft_consensus.cc:740] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3517f3498e5e4e9cb5cbfa239e4999e3, State: Initialized, Role: FOLLOWER
I20260504 13:57:32.050235 13076 consensus_queue.cc:260] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:32.051093 13075 ts_tablet_manager.cc:1929] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:57:32.051542 13075 log.cc:1199] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/a9cfb4321395442891167633a8c76dea
I20260504 13:57:32.052426 13075 ts_tablet_manager.cc:1950] T a9cfb4321395442891167633a8c76dea P 3517f3498e5e4e9cb5cbfa239e4999e3: Deleting consensus metadata
I20260504 13:57:32.053032 13072 tablet_bootstrap.cc:492] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6: No bootstrap required, opened a new log
I20260504 13:57:32.054839 13076 ts_tablet_manager.cc:1434] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260504 13:57:32.053457 13072 ts_tablet_manager.cc:1403] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.002s
I20260504 13:57:32.061597 13078 ts_tablet_manager.cc:1929] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:57:32.062016 13078 log.cc:1199] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/4c771aa2ad854224b235afb92cbb5ea5
I20260504 13:57:32.062704 13078 ts_tablet_manager.cc:1950] T 4c771aa2ad854224b235afb92cbb5ea5 P 7c95ca5995cc4cbba624fd3ea6367902: Deleting consensus metadata
I20260504 13:57:32.063771 13072 raft_consensus.cc:359] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:32.064587 13072 raft_consensus.cc:385] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:32.064916 13072 raft_consensus.cc:740] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910fbf5d51f843609f5caf25e52beae6, State: Initialized, Role: FOLLOWER
I20260504 13:57:32.065569 13072 consensus_queue.cc:260] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:32.068948 13075 tablet_replica.cc:333] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3: stopping tablet replica
I20260504 13:57:32.069461 13072 ts_tablet_manager.cc:1434] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260504 13:57:32.072247 13075 raft_consensus.cc:2243] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:57:32.072718 13075 raft_consensus.cc:2272] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260504 13:57:32.073094 13018 tablet.cc:2404] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260504 13:57:32.074858 13075 ts_tablet_manager.cc:1916] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:57:32.097895 13075 ts_tablet_manager.cc:1929] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:57:32.098233 13075 log.cc:1199] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/87d2ae9451964160836b7d6aa6d95587
I20260504 13:57:32.098851 13075 ts_tablet_manager.cc:1950] T 87d2ae9451964160836b7d6aa6d95587 P 3517f3498e5e4e9cb5cbfa239e4999e3: Deleting consensus metadata
I20260504 13:57:32.101423 13075 tablet_replica.cc:333] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3: stopping tablet replica
I20260504 13:57:32.102092 13075 raft_consensus.cc:2243] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260504 13:57:32.102517 13075 raft_consensus.cc:2272] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260504 13:57:32.105262 13075 ts_tablet_manager.cc:1916] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:57:32.118252 13075 ts_tablet_manager.cc:1929] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260504 13:57:32.118604 13075 log.cc:1199] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/4c771aa2ad854224b235afb92cbb5ea5
I20260504 13:57:32.119191 13075 ts_tablet_manager.cc:1950] T 4c771aa2ad854224b235afb92cbb5ea5 P 3517f3498e5e4e9cb5cbfa239e4999e3: Deleting consensus metadata
W20260504 13:57:32.227316 13080 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260504 13:57:32.228194 13080 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260504 13:57:32.228899 13080 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260504 13:57:32.249228 13084 raft_consensus.cc:493] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:32.249665 13084 raft_consensus.cc:515] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:32.253837 13084 leader_election.cc:290] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331), 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229)
I20260504 13:57:32.278342 12675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6388ce6c7f49db91576194fe741a73" candidate_uuid: "7c95ca5995cc4cbba624fd3ea6367902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910fbf5d51f843609f5caf25e52beae6" is_pre_election: true
I20260504 13:57:32.279196 12675 raft_consensus.cc:2468] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c95ca5995cc4cbba624fd3ea6367902 in term 0.
I20260504 13:57:32.280426 12900 leader_election.cc:304] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c95ca5995cc4cbba624fd3ea6367902, 910fbf5d51f843609f5caf25e52beae6; no voters:
I20260504 13:57:32.281276 13084 raft_consensus.cc:2804] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:57:32.281625 13084 raft_consensus.cc:493] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:57:32.281939 13084 raft_consensus.cc:3060] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:57:32.283200 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6388ce6c7f49db91576194fe741a73" candidate_uuid: "7c95ca5995cc4cbba624fd3ea6367902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" is_pre_election: true
I20260504 13:57:32.283985 12818 raft_consensus.cc:2468] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c95ca5995cc4cbba624fd3ea6367902 in term 0.
I20260504 13:57:32.288873 13084 raft_consensus.cc:515] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
W20260504 13:57:32.289793 13080 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260504 13:57:32.290130 13080 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260504 13:57:32.290716 13080 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260504 13:57:32.291491 13080 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
I20260504 13:57:32.291440 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6388ce6c7f49db91576194fe741a73" candidate_uuid: "7c95ca5995cc4cbba624fd3ea6367902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3"
W20260504 13:57:32.291882 13080 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
I20260504 13:57:32.291966 12818 raft_consensus.cc:3060] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:57:32.292235 13080 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260504 13:57:32.299489 12818 raft_consensus.cc:2468] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c95ca5995cc4cbba624fd3ea6367902 in term 1.
I20260504 13:57:32.300647 12901 leader_election.cc:304] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3517f3498e5e4e9cb5cbfa239e4999e3, 7c95ca5995cc4cbba624fd3ea6367902; no voters:
I20260504 13:57:32.307034 13084 leader_election.cc:290] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [CANDIDATE]: Term 1 election: Requested vote from peers 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331), 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229)
I20260504 13:57:32.307336 12675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f6388ce6c7f49db91576194fe741a73" candidate_uuid: "7c95ca5995cc4cbba624fd3ea6367902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910fbf5d51f843609f5caf25e52beae6"
I20260504 13:57:32.307619 13084 raft_consensus.cc:2804] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:57:32.308032 13084 raft_consensus.cc:697] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [term 1 LEADER]: Becoming Leader. State: Replica: 7c95ca5995cc4cbba624fd3ea6367902, State: Running, Role: LEADER
I20260504 13:57:32.307878 12675 raft_consensus.cc:3060] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:57:32.308887 13084 consensus_queue.cc:237] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:32.323553 12675 raft_consensus.cc:2468] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c95ca5995cc4cbba624fd3ea6367902 in term 1.
I20260504 13:57:32.334026 13118 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bc3d546912c9427aaca93bdafc33dd6d)
I20260504 13:57:32.334627 13118 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:32.336441 13118 leader_election.cc:290] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
W20260504 13:57:32.359992 12371 leader_election.cc:336] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825): Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:32.376200 13080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.238.212:41493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:39899
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.252:35825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.238.252
--webserver_port=35067
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 04 May 2026 13:43:23 UTC on e7f111948823
build id 11743
TSAN enabled
I20260504 13:57:32.378058 13080 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 13:57:32.380378 13080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 13:57:32.394474 13124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:57:32.401476 12752 connection.cc:570] server connection from 127.0.0.1:49474 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:32.402808 13023 connection.cc:570] client connection to 127.8.238.254:38403 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:32.407069 13023 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.238.254:38403)
W20260504 13:57:32.408932 13023 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.8.238.254:38403)
W20260504 13:57:32.410550 13023 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.238.254:38403)
W20260504 13:57:32.414047 13023 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.238.254:38403)
W20260504 13:57:32.423768 12753 connection.cc:570] client connection to 127.8.238.254:38403 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:32.426718 12902 connection.cc:570] server connection from 127.0.0.1:44786 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:32.427958 12610 connection.cc:570] client connection to 127.8.238.254:38403 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:32.432525 12611 connection.cc:570] server connection from 127.0.0.1:47760 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:32.434235 12370 connection.cc:570] client connection to 127.8.238.254:38403 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:32.438462 13073 negotiation.cc:336] Failed RPC negotiation. Trace:
0504 13:57:32.421534 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.8.238.254:38403 (local address 127.8.238.195:51331)
0504 13:57:32.422813 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0504 13:57:32.422874 (+ 61us) client_negotiation.cc:175] Beginning negotiation
0504 13:57:32.423218 (+ 344us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0504 13:57:32.436989 (+ 13771us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":416}
W20260504 13:57:32.439898 13129 negotiation.cc:336] Failed RPC negotiation. Trace:
0504 13:57:32.418493 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.8.238.254:38403 (local address 127.0.0.1:55106)
0504 13:57:32.437334 (+ 18841us) negotiation.cc:107] Waiting for socket to connect
0504 13:57:32.437642 (+ 308us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":13050,"threads_started":1}
W20260504 13:57:32.441461 13089 negotiation.cc:336] Failed RPC negotiation. Trace:
0504 13:57:32.425922 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.8.238.254:38403 (local address 127.8.238.194:35497)
0504 13:57:32.426710 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0504 13:57:32.426755 (+ 45us) client_negotiation.cc:175] Beginning negotiation
0504 13:57:32.427008 (+ 253us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0504 13:57:32.440436 (+ 13428us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":76}
W20260504 13:57:32.441613 13062 negotiation.cc:336] Failed RPC negotiation. Trace:
0504 13:57:32.430170 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.8.238.254:38403 (local address 127.8.238.193:45661)
0504 13:57:32.431511 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0504 13:57:32.431564 (+ 53us) client_negotiation.cc:175] Beginning negotiation
0504 13:57:32.431889 (+ 325us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0504 13:57:32.440615 (+ 8726us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":400}
W20260504 13:57:32.444932 13121 negotiation.cc:336] Failed RPC negotiation. Trace:
0504 13:57:32.436014 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.8.238.254:38403 (local address 127.0.0.1:55108)
0504 13:57:32.436688 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0504 13:57:32.436726 (+ 38us) client_negotiation.cc:175] Beginning negotiation
0504 13:57:32.437047 (+ 321us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0504 13:57:32.443501 (+ 6454us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":92}
W20260504 13:57:32.445674 12370 leader_election.cc:336] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260504 13:57:32.446358 12370 leader_election.cc:304] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters: 26e2451840464331a15d5291c3f09034, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:32.447402 13118 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 13:57:32.998549 13130 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bc3d546912c9427aaca93bdafc33dd6d)
I20260504 13:57:32.999176 13130 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:33.005903 13130 leader_election.cc:290] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
W20260504 13:57:33.022982 12370 leader_election.cc:336] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:57:33.023468 12371 leader_election.cc:336] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825): Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
I20260504 13:57:33.023877 12371 leader_election.cc:304] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters: 26e2451840464331a15d5291c3f09034, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:33.025357 13130 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260504 13:57:33.349524 13135 consensus_queue.cc:1048] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:57:33.412214 13135 consensus_queue.cc:1048] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260504 13:57:33.808799 13147 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bc3d546912c9427aaca93bdafc33dd6d)
I20260504 13:57:33.809455 13147 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:33.817945 13147 leader_election.cc:290] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
W20260504 13:57:33.839954 12371 leader_election.cc:336] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825): Network error: Client connection negotiation failed: client connection to 127.8.238.252:35825: connect: Connection refused (error 111)
W20260504 13:57:33.843209 12370 leader_election.cc:336] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
I20260504 13:57:33.843694 12370 leader_election.cc:304] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters: 26e2451840464331a15d5291c3f09034, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:33.844415 13147 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260504 13:57:33.796028 13123 debug-util.cc:398] Leaking SignalData structure 0x7b080002ecc0 after lost signal to thread 13080
W20260504 13:57:32.397492 13125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:57:33.864136 13080 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.469s user 0.479s sys 0.944s
W20260504 13:57:33.864639 13080 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.469s user 0.480s sys 0.944s
W20260504 13:57:33.872871 13151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:57:33.873232 13080 server_base.cc:1061] running on GCE node
I20260504 13:57:33.874454 13080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260504 13:57:33.876880 13080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260504 13:57:33.878271 13080 hybrid_clock.cc:648] HybridClock initialized: now 1777903053878236 us; error 30 us; skew 500 ppm
I20260504 13:57:33.879258 13080 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 13:57:33.886075 13080 webserver.cc:492] Webserver started at http://127.8.238.252:35067/ using document root <none> and password file <none>
I20260504 13:57:33.887336 13080 fs_manager.cc:362] Metadata directory not provided
I20260504 13:57:33.887609 13080 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 13:57:33.895733 13080 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260504 13:57:33.900787 13157 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 13:57:33.902089 13080 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260504 13:57:33.902459 13080 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data,/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal
uuid: "26e2451840464331a15d5291c3f09034"
format_stamp: "Formatted at 2026-05-04 13:57:21 on dist-test-slave-f9wt"
I20260504 13:57:33.904912 13080 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 13:57:33.977332 13080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 13:57:33.978847 13080 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 13:57:33.979394 13080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 13:57:34.051949 13080 rpc_server.cc:307] RPC server started. Bound to: 127.8.238.252:35825
I20260504 13:57:34.052026 13210 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.238.252:35825 every 8 connection(s)
I20260504 13:57:34.054764 13080 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data/info.pb
I20260504 13:57:34.060086 9147 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu as pid 13080
I20260504 13:57:34.071774 13211 sys_catalog.cc:263] Verifying existing consensus state
I20260504 13:57:34.079232 13211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: Bootstrap starting.
I20260504 13:57:34.095080 13211 log.cc:826] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: Log is configured to *not* fsync() on all Append() calls
I20260504 13:57:34.117259 13211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 13:57:34.118165 13211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: Bootstrap complete.
I20260504 13:57:34.137552 13211 raft_consensus.cc:359] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:34.140334 13211 raft_consensus.cc:740] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e2451840464331a15d5291c3f09034, State: Initialized, Role: FOLLOWER
I20260504 13:57:34.141148 13211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:34.144301 13215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:34.144985 13211 sys_catalog.cc:565] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: configured and running, proceeding with master startup.
I20260504 13:57:34.145172 13215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:57:34.177968 13230 catalog_manager.cc:1269] Loaded cluster ID: 8a9361cd032446ef9af554d6cfd5f3e8
I20260504 13:57:34.178481 13230 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: success
I20260504 13:57:34.195042 13230 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: acquiring CA information for follower catalog manager: success
I20260504 13:57:34.201084 13230 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260504 13:57:34.194953 13080 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:57:34.214027 13080 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:57:34.422119 12721 heartbeater.cc:344] Connected to a master server at 127.8.238.252:35825
I20260504 13:57:34.429653 13176 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" instance_seqno: 1777903046020339) as {username='slave'} at 127.8.238.193:57199; Asking this server to re-register.
I20260504 13:57:34.431452 12721 heartbeater.cc:461] Registering TS with master...
I20260504 13:57:34.435556 13175 ts_manager.cc:194] Registered new tserver with Master: 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229)
I20260504 13:57:34.440207 13011 heartbeater.cc:344] Connected to a master server at 127.8.238.252:35825
I20260504 13:57:34.440702 13011 heartbeater.cc:499] Master 127.8.238.252:35825 was elected leader, sending a full tablet report...
I20260504 13:57:34.444535 13175 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" instance_seqno: 1777903050251388) as {username='slave'} at 127.8.238.195:56479; Asking this server to re-register.
I20260504 13:57:34.446130 13011 heartbeater.cc:461] Registering TS with master...
I20260504 13:57:34.448788 13175 ts_manager.cc:194] Registered new tserver with Master: 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223)
I20260504 13:57:34.452726 13239 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bc3d546912c9427aaca93bdafc33dd6d)
I20260504 13:57:34.453208 13239 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:34.462913 13239 leader_election.cc:290] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
W20260504 13:57:34.470028 12370 leader_election.cc:336] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
I20260504 13:57:34.474012 13186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034" is_pre_election: true
I20260504 13:57:34.475095 13186 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc832d2f7f748dbb8e3ba2f73ec98a4 in term 1.
I20260504 13:57:34.476718 12371 leader_election.cc:304] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26e2451840464331a15d5291c3f09034, 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters: bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:34.477713 13239 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260504 13:57:34.478027 13239 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bc3d546912c9427aaca93bdafc33dd6d)
I20260504 13:57:34.478348 13239 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 2 FOLLOWER]: Advancing to term 3
I20260504 13:57:34.478394 12863 heartbeater.cc:344] Connected to a master server at 127.8.238.252:35825
I20260504 13:57:34.481796 13175 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" instance_seqno: 1777903048076284) as {username='slave'} at 127.8.238.194:35099; Asking this server to re-register.
I20260504 13:57:34.483080 12863 heartbeater.cc:461] Registering TS with master...
I20260504 13:57:34.484894 13239 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:34.486789 13175 ts_manager.cc:194] Registered new tserver with Master: 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331)
I20260504 13:57:34.487409 13239 leader_election.cc:290] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 election: Requested vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 26e2451840464331a15d5291c3f09034 (127.8.238.252:35825)
I20260504 13:57:34.487788 13186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "26e2451840464331a15d5291c3f09034"
I20260504 13:57:34.488303 13186 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 1 FOLLOWER]: Advancing to term 3
W20260504 13:57:34.490729 12370 leader_election.cc:336] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
I20260504 13:57:34.496007 13186 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc832d2f7f748dbb8e3ba2f73ec98a4 in term 3.
I20260504 13:57:34.496884 12371 leader_election.cc:304] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26e2451840464331a15d5291c3f09034, 6dc832d2f7f748dbb8e3ba2f73ec98a4; no voters: bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:34.497498 13239 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 3 FOLLOWER]: Leader election won for term 3
I20260504 13:57:34.498785 13239 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [term 3 LEADER]: Becoming Leader. State: Replica: 6dc832d2f7f748dbb8e3ba2f73ec98a4, State: Running, Role: LEADER
I20260504 13:57:34.499588 13239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 14, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:34.503388 13243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dc832d2f7f748dbb8e3ba2f73ec98a4. Latest consensus state: current_term: 3 leader_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:34.503899 13243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [sys.catalog]: This master's current role is: LEADER
I20260504 13:57:34.505966 13245 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260504 13:57:34.514950 13245 catalog_manager.cc:679] Loaded metadata for table default.table_76e3f88f42b84f1397e06f486e84d95a [id=8b7a3413a9f44a8c915ffa9ff2484c41]
I20260504 13:57:34.516737 13245 catalog_manager.cc:679] Loaded metadata for table default.table_741fc94d99de4e8ab34684aa3f168034 [id=ab5674daf2194017b802477a5586ec14]
I20260504 13:57:34.518411 13245 catalog_manager.cc:679] Loaded metadata for table default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614]
I20260504 13:57:34.524646 13245 tablet_loader.cc:96] loaded metadata for tablet 2de1fe3c24ba40088ccb83968db0cf76 (table default.table_741fc94d99de4e8ab34684aa3f168034 [id=ab5674daf2194017b802477a5586ec14])
I20260504 13:57:34.526928 13245 tablet_loader.cc:96] loaded metadata for tablet 6f6388ce6c7f49db91576194fe741a73 (table default.table_76e3f88f42b84f1397e06f486e84d95a [id=8b7a3413a9f44a8c915ffa9ff2484c41])
I20260504 13:57:34.528900 13245 tablet_loader.cc:96] loaded metadata for tablet 8896d8df10394e30ac15f3b04625b13b (table default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614])
I20260504 13:57:34.531061 13245 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260504 13:57:34.535287 13245 catalog_manager.cc:1269] Loaded cluster ID: 8a9361cd032446ef9af554d6cfd5f3e8
I20260504 13:57:34.535537 13245 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260504 13:57:34.540417 13245 catalog_manager.cc:1514] Loading token signing keys...
I20260504 13:57:34.544415 13245 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4: Loaded TSK: 0
I20260504 13:57:34.545641 13245 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260504 13:57:34.549615 13245 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260504 13:57:34.553434 13245 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260504 13:57:34.568015 13186 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Refusing update from remote peer 6dc832d2f7f748dbb8e3ba2f73ec98a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20260504 13:57:34.569514 13239 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 4, Time since last communication: 0.000s
W20260504 13:57:34.570240 12370 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 -> Peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Couldn't send request to peer bc3d546912c9427aaca93bdafc33dd6d. Status: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260504 13:57:34.599834 12383 master_service.cc:954] ReplaceTablet: received request to replace tablet 6f6388ce6c7f49db91576194fe741a73 from {username='slave'} at 127.0.0.1:52014
I20260504 13:57:34.600106 12384 master_service.cc:954] ReplaceTablet: received request to replace tablet 6f6388ce6c7f49db91576194fe741a73 from {username='slave'} at 127.0.0.1:52014
I20260504 13:57:34.601064 12382 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52014:
table { table_name: "default.Table_741fc94d99de4e8ab34684aa3f168034" } modify_external_catalogs: true
I20260504 13:57:34.601851 12382 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52014:
table { table_name: "default.Table_741fc94d99de4e8ab34684aa3f168034" } modify_external_catalogs: true
I20260504 13:57:34.604337 12381 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52014:
name: "default.Table_398fe552fc9a43ac8875babd6d5001fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:57:34.612234 12380 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52014:
name: "default.Table_2dc3318442a1464eb3c4db4f57a86c83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260504 13:57:34.613353 12540 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1777903050, eventType=CREATE_TABLE, dbName=default, tableName=table_76e3f88f42b84f1397e06f486e84d95a, message={"server":"","servicePrincipal":"","db":"default","table":"table_76e3f88f42b84f1397e06f486e84d95a","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_76e3f88f42b84f1397e06f486e84d95a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777903050},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/warehouse/table_76e3f88f42b84f1397e06f486e84d95a\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_76e3f88f42b84f1397e06f486e84d95a\",\"kudu.table_id\":\"8b7a3413a9f44a8c915ffa9ff2484c41\",\"transient_lastDdlTime\":\"1777903050\",\"comment\":\"\",\"kudu.master_addresses\":\"127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8a9361cd032446ef9af554d6cfd5f3e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777903050,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1777903050, eventType=CREATE_TABLE, dbName=default, tableName=table_53b74b33cd8b42e883e98214b6266851, message={"server":"","servicePrincipal":"","db":"default","table":"table_53b74b33cd8b42e883e98214b6266851","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_53b74b33cd8b42e883e98214b6266851\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1777903050},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/warehouse/table_53b74b33cd8b42e883e98214b6266851\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_53b74b33cd8b42e883e98214b6266851\",\"kudu.table_id\":\"389bb89032254cbd88075a6da721baf6\",\"transient_lastDdlTime\":\"1777903050\",\"comment\":\"\",\"kudu.master_addresses\":\"127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"8a9361cd032446ef9af554d6cfd5f3e8\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1777903050,"files":[]}, messageFormat=json-0.2, catName=hive);
I20260504 13:57:34.614624 12379 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52014:
name: "default.Table_91bfcb8f6cb843cea4deee25d923b07c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
*** Check failure stack trace: ***
I20260504 13:57:34.616060 12378 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52014:
name: "default.Table_8366b20cb82349ea8ee8c637ca836adf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260504 13:57:34.617933 12377 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52014:
table { table_name: "default.Table_4144f4abff494ea3b2fffc68d535e514" } new_table_name: "default.Table_c4220efa2c914d1391eb4d7f5459d3c3" modify_external_catalogs: true
I20260504 13:57:34.621071 12376 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52014:
table { table_name: "default.Table_76e3f88f42b84f1397e06f486e84d95a" } new_table_name: "default.Table_14adc72f313248df95e58d7dfaa02df9" modify_external_catalogs: true
I20260504 13:57:34.628319 13246 mvcc.cc:204] Tried to move back new op lower bound from 7282290911496138752 to 7282290911236714496. Current Snapshot: MvccSnapshot[applied={T|T < 7282290911496138752}]
I20260504 13:57:34.628456 13215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dc832d2f7f748dbb8e3ba2f73ec98a4. Latest consensus state: current_term: 3 leader_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:34.629194 13215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:57:34.633514 13249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:34.634379 13249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:57:34.645238 13239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:34.645928 13239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4 [sys.catalog]: This master's current role is: LEADER
I20260504 13:57:34.649350 13253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:34.653012 13253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f933b4da950 google::LogMessage::Flush() at ??:0
I20260504 13:57:34.677265 13247 mvcc.cc:204] Tried to move back new op lower bound from 7282290911496138752 to 7282290895886381056. Current Snapshot: MvccSnapshot[applied={T|T < 7282290911496138752 or (T in {7282290911496138752})}]
I20260504 13:57:34.693640 12383 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4: Sending DeleteTablet for 3 replicas of tablet 6f6388ce6c7f49db91576194fe741a73
I20260504 13:57:34.695573 12383 catalog_manager.cc:6741] ReplaceTablet: tablet 6f6388ce6c7f49db91576194fe741a73 deleted and replaced by tablet aa4bb234488c470a9ef680243cd09522
@ 0x7f933b4da6bf google::LogMessage::~LogMessage() at ??:0
@ 0x7f934659a2bc kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260504 13:57:34.746333 12655 tablet_service.cc:1511] Processing CreateTablet for tablet 8896d8df10394e30ac15f3b04625b13b (DEFAULT_TABLE table=default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:34.747773 12655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8896d8df10394e30ac15f3b04625b13b. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f9346599a7c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260504 13:57:34.762313 12945 tablet_service.cc:1511] Processing CreateTablet for tablet 8896d8df10394e30ac15f3b04625b13b (DEFAULT_TABLE table=default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:34.763398 12945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8896d8df10394e30ac15f3b04625b13b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:34.776942 12654 tablet_service.cc:1511] Processing CreateTablet for tablet 2de1fe3c24ba40088ccb83968db0cf76 (DEFAULT_TABLE table=default.table_741fc94d99de4e8ab34684aa3f168034 [id=ab5674daf2194017b802477a5586ec14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:34.777431 12946 tablet_service.cc:1511] Processing CreateTablet for tablet 2de1fe3c24ba40088ccb83968db0cf76 (DEFAULT_TABLE table=default.table_741fc94d99de4e8ab34684aa3f168034 [id=ab5674daf2194017b802477a5586ec14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:34.778158 12654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de1fe3c24ba40088ccb83968db0cf76. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f934659daf2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260504 13:57:34.778508 12946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de1fe3c24ba40088ccb83968db0cf76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:34.792011 12653 tablet_service.cc:1558] Processing DeleteTablet for tablet 6f6388ce6c7f49db91576194fe741a73 with delete_type TABLET_DATA_DELETED (replaced by tablet aa4bb234488c470a9ef680243cd09522) from {username='slave'} at 127.0.0.1:58062
I20260504 13:57:34.799011 12944 tablet_service.cc:1558] Processing DeleteTablet for tablet 6f6388ce6c7f49db91576194fe741a73 with delete_type TABLET_DATA_DELETED (replaced by tablet aa4bb234488c470a9ef680243cd09522) from {username='slave'} at 127.0.0.1:41360
I20260504 13:57:34.802716 13267 tablet_bootstrap.cc:492] T 2de1fe3c24ba40088ccb83968db0cf76 P 7c95ca5995cc4cbba624fd3ea6367902: Bootstrap starting.
I20260504 13:57:34.803858 13268 tablet_replica.cc:333] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902: stopping tablet replica
@ 0x7f934659daaa _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260504 13:57:34.806628 13268 raft_consensus.cc:2243] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [term 1 LEADER]: Raft consensus shutting down.
I20260504 13:57:34.807689 13268 raft_consensus.cc:2272] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:57:34.808990 13267 tablet_bootstrap.cc:654] T 2de1fe3c24ba40088ccb83968db0cf76 P 7c95ca5995cc4cbba624fd3ea6367902: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:34.809648 13268 ts_tablet_manager.cc:1916] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:57:34.810871 12652 tablet_service.cc:1511] Processing CreateTablet for tablet aa4bb234488c470a9ef680243cd09522 (DEFAULT_TABLE table=default.table_76e3f88f42b84f1397e06f486e84d95a [id=8b7a3413a9f44a8c915ffa9ff2484c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:34.812057 12652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa4bb234488c470a9ef680243cd09522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:34.821579 13267 tablet_bootstrap.cc:492] T 2de1fe3c24ba40088ccb83968db0cf76 P 7c95ca5995cc4cbba624fd3ea6367902: No bootstrap required, opened a new log
I20260504 13:57:34.821964 13267 ts_tablet_manager.cc:1403] T 2de1fe3c24ba40088ccb83968db0cf76 P 7c95ca5995cc4cbba624fd3ea6367902: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.007s
I20260504 13:57:34.824666 13267 raft_consensus.cc:359] T 2de1fe3c24ba40088ccb83968db0cf76 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:34.825686 13267 raft_consensus.cc:385] T 2de1fe3c24ba40088ccb83968db0cf76 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:34.825954 13267 raft_consensus.cc:740] T 2de1fe3c24ba40088ccb83968db0cf76 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c95ca5995cc4cbba624fd3ea6367902, State: Initialized, Role: FOLLOWER
I20260504 13:57:34.826776 13267 consensus_queue.cc:260] T 2de1fe3c24ba40088ccb83968db0cf76 P 7c95ca5995cc4cbba624fd3ea6367902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:34.828725 13267 ts_tablet_manager.cc:1434] T 2de1fe3c24ba40088ccb83968db0cf76 P 7c95ca5995cc4cbba624fd3ea6367902: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260504 13:57:34.838169 13012 heartbeater.cc:499] Master 127.8.238.253:35717 was elected leader, sending a full tablet report...
I20260504 13:57:34.842851 13266 tablet_replica.cc:333] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6: stopping tablet replica
I20260504 13:57:34.843026 12796 tablet_service.cc:1511] Processing CreateTablet for tablet 8896d8df10394e30ac15f3b04625b13b (DEFAULT_TABLE table=default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:34.844105 12796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8896d8df10394e30ac15f3b04625b13b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:34.846791 12946 tablet_service.cc:1511] Processing CreateTablet for tablet aa4bb234488c470a9ef680243cd09522 (DEFAULT_TABLE table=default.table_76e3f88f42b84f1397e06f486e84d95a [id=8b7a3413a9f44a8c915ffa9ff2484c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:34.847862 12946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa4bb234488c470a9ef680243cd09522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:34.851559 13268 ts_tablet_manager.cc:1929] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:57:34.853575 13266 raft_consensus.cc:2243] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:57:34.854190 13266 raft_consensus.cc:2272] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f934659da3a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260504 13:57:34.858422 13268 log.cc:1199] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-2/wal/wals/6f6388ce6c7f49db91576194fe741a73
I20260504 13:57:34.859243 13268 ts_tablet_manager.cc:1950] T 6f6388ce6c7f49db91576194fe741a73 P 7c95ca5995cc4cbba624fd3ea6367902: Deleting consensus metadata
I20260504 13:57:34.862004 12370 catalog_manager.cc:5002] TS 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223): tablet 6f6388ce6c7f49db91576194fe741a73 (table default.table_76e3f88f42b84f1397e06f486e84d95a [id=8b7a3413a9f44a8c915ffa9ff2484c41]) successfully deleted
I20260504 13:57:34.862542 12797 tablet_service.cc:1511] Processing CreateTablet for tablet 2de1fe3c24ba40088ccb83968db0cf76 (DEFAULT_TABLE table=default.table_741fc94d99de4e8ab34684aa3f168034 [id=ab5674daf2194017b802477a5586ec14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:34.863601 12797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de1fe3c24ba40088ccb83968db0cf76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:34.863601 13267 tablet_bootstrap.cc:492] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902: Bootstrap starting.
I20260504 13:57:34.865769 12795 tablet_service.cc:1511] Processing CreateTablet for tablet aa4bb234488c470a9ef680243cd09522 (DEFAULT_TABLE table=default.table_76e3f88f42b84f1397e06f486e84d95a [id=8b7a3413a9f44a8c915ffa9ff2484c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:34.866832 12795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa4bb234488c470a9ef680243cd09522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:34.867389 12798 tablet_service.cc:1558] Processing DeleteTablet for tablet 6f6388ce6c7f49db91576194fe741a73 with delete_type TABLET_DATA_DELETED (replaced by tablet aa4bb234488c470a9ef680243cd09522) from {username='slave'} at 127.0.0.1:39496
@ 0x7f934659da02 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260504 13:57:34.888022 13266 ts_tablet_manager.cc:1916] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:57:34.896385 13272 tablet_bootstrap.cc:492] T 8896d8df10394e30ac15f3b04625b13b P 910fbf5d51f843609f5caf25e52beae6: Bootstrap starting.
I20260504 13:57:34.902769 12384 master_service.cc:954] ReplaceTablet: received request to replace tablet 8896d8df10394e30ac15f3b04625b13b from {username='slave'} at 127.0.0.1:52014
@ 0x7f934659ccfe std::__1::__function::__func<>::operator()() at ??:0
@ 0x5b3bc5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5b39fa std::__1::function<>::operator()() at ??:?
I20260504 13:57:34.921563 12384 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6dc832d2f7f748dbb8e3ba2f73ec98a4: Sending DeleteTablet for 3 replicas of tablet 8896d8df10394e30ac15f3b04625b13b
I20260504 13:57:34.925093 12945 tablet_service.cc:1558] Processing DeleteTablet for tablet 8896d8df10394e30ac15f3b04625b13b with delete_type TABLET_DATA_DELETED (replaced by tablet e436ef80b9d845c0a313fd7e95229bd1) from {username='slave'} at 127.0.0.1:41360
I20260504 13:57:34.926429 12383 master_service.cc:954] ReplaceTablet: received request to replace tablet 8896d8df10394e30ac15f3b04625b13b from {username='slave'} at 127.0.0.1:52014
W20260504 13:57:34.928356 12370 catalog_manager.cc:4982] TS 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223): delete failed for tablet 8896d8df10394e30ac15f3b04625b13b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8896d8df10394e30ac15f3b04625b13b already in progress: creating tablet
I20260504 13:57:34.923018 12384 catalog_manager.cc:6741] ReplaceTablet: tablet 8896d8df10394e30ac15f3b04625b13b deleted and replaced by tablet e436ef80b9d845c0a313fd7e95229bd1
I20260504 13:57:34.931881 12798 tablet_service.cc:1558] Processing DeleteTablet for tablet 8896d8df10394e30ac15f3b04625b13b with delete_type TABLET_DATA_DELETED (replaced by tablet e436ef80b9d845c0a313fd7e95229bd1) from {username='slave'} at 127.0.0.1:39496
I20260504 13:57:34.938979 13267 tablet_bootstrap.cc:654] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:34.948185 12652 tablet_service.cc:1558] Processing DeleteTablet for tablet 8896d8df10394e30ac15f3b04625b13b with delete_type TABLET_DATA_DELETED (replaced by tablet e436ef80b9d845c0a313fd7e95229bd1) from {username='slave'} at 127.0.0.1:58062
I20260504 13:57:34.955315 12652 tablet_service.cc:1511] Processing CreateTablet for tablet e436ef80b9d845c0a313fd7e95229bd1 (DEFAULT_TABLE table=default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:34.955663 12798 tablet_service.cc:1511] Processing CreateTablet for tablet e436ef80b9d845c0a313fd7e95229bd1 (DEFAULT_TABLE table=default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:34.956696 12652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e436ef80b9d845c0a313fd7e95229bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:34.957617 12798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e436ef80b9d845c0a313fd7e95229bd1. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f933d700fbb kudu::Thread::SuperviseThread() at ??:0
@ 0x479510 __tsan_thread_start_func at ??:?
I20260504 13:57:34.960006 12945 tablet_service.cc:1511] Processing CreateTablet for tablet e436ef80b9d845c0a313fd7e95229bd1 (DEFAULT_TABLE table=default.table_4144f4abff494ea3b2fffc68d535e514 [id=be3e1aa95f514f9db45c7887bc578614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260504 13:57:34.961127 12945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e436ef80b9d845c0a313fd7e95229bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260504 13:57:34.970387 13273 tablet_replica.cc:333] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3: stopping tablet replica
I20260504 13:57:34.971064 13273 raft_consensus.cc:2243] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260504 13:57:34.971628 13273 raft_consensus.cc:2272] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260504 13:57:34.973654 13273 ts_tablet_manager.cc:1916] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260504 13:57:34.976609 13272 tablet_bootstrap.cc:654] T 8896d8df10394e30ac15f3b04625b13b P 910fbf5d51f843609f5caf25e52beae6: Neither blocks nor log segments found. Creating new log.
@ 0x7f933edfc6db start_thread at ??:0
I20260504 13:57:34.986060 13275 tablet_bootstrap.cc:492] T aa4bb234488c470a9ef680243cd09522 P 3517f3498e5e4e9cb5cbfa239e4999e3: Bootstrap starting.
I20260504 13:57:34.996223 13266 ts_tablet_manager.cc:1929] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:57:35.003280 13266 log.cc:1199] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-0/wal/wals/6f6388ce6c7f49db91576194fe741a73
@ 0x7f933a17f61f clone at ??:0
I20260504 13:57:35.004688 13266 ts_tablet_manager.cc:1950] T 6f6388ce6c7f49db91576194fe741a73 P 910fbf5d51f843609f5caf25e52beae6: Deleting consensus metadata
I20260504 13:57:35.008025 13272 tablet_bootstrap.cc:492] T 8896d8df10394e30ac15f3b04625b13b P 910fbf5d51f843609f5caf25e52beae6: No bootstrap required, opened a new log
I20260504 13:57:35.008433 13272 ts_tablet_manager.cc:1403] T 8896d8df10394e30ac15f3b04625b13b P 910fbf5d51f843609f5caf25e52beae6: Time spent bootstrapping tablet: real 0.112s user 0.011s sys 0.000s
I20260504 13:57:35.010454 13275 tablet_bootstrap.cc:654] T aa4bb234488c470a9ef680243cd09522 P 3517f3498e5e4e9cb5cbfa239e4999e3: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:35.012562 13267 tablet_bootstrap.cc:492] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902: No bootstrap required, opened a new log
I20260504 13:57:35.012970 13267 ts_tablet_manager.cc:1403] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902: Time spent bootstrapping tablet: real 0.150s user 0.004s sys 0.008s
I20260504 13:57:35.014858 13272 raft_consensus.cc:359] T 8896d8df10394e30ac15f3b04625b13b P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } }
I20260504 13:57:35.015636 13272 raft_consensus.cc:385] T 8896d8df10394e30ac15f3b04625b13b P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:35.015532 13267 raft_consensus.cc:359] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } }
I20260504 13:57:35.015933 13272 raft_consensus.cc:740] T 8896d8df10394e30ac15f3b04625b13b P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910fbf5d51f843609f5caf25e52beae6, State: Initialized, Role: FOLLOWER
I20260504 13:57:35.016155 13267 raft_consensus.cc:385] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:35.016417 13267 raft_consensus.cc:740] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c95ca5995cc4cbba624fd3ea6367902, State: Initialized, Role: FOLLOWER
I20260504 13:57:35.017197 13267 consensus_queue.cc:260] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } }
I20260504 13:57:35.018232 13272 consensus_queue.cc:260] T 8896d8df10394e30ac15f3b04625b13b P 910fbf5d51f843609f5caf25e52beae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } }
I20260504 13:57:35.022114 13273 ts_tablet_manager.cc:1929] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260504 13:57:35.022473 13273 log.cc:1199] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3: Deleting WAL directory at /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/ts-1/wal/wals/6f6388ce6c7f49db91576194fe741a73
I20260504 13:57:35.023288 13273 ts_tablet_manager.cc:1950] T 6f6388ce6c7f49db91576194fe741a73 P 3517f3498e5e4e9cb5cbfa239e4999e3: Deleting consensus metadata
I20260504 13:57:35.028901 13275 tablet_bootstrap.cc:492] T aa4bb234488c470a9ef680243cd09522 P 3517f3498e5e4e9cb5cbfa239e4999e3: No bootstrap required, opened a new log
I20260504 13:57:35.029305 13275 ts_tablet_manager.cc:1403] T aa4bb234488c470a9ef680243cd09522 P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.000s
I20260504 13:57:35.031356 13267 ts_tablet_manager.cc:1434] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20260504 13:57:35.031863 12722 heartbeater.cc:499] Master 127.8.238.253:35717 was elected leader, sending a full tablet report...
I20260504 13:57:35.039446 13275 raft_consensus.cc:359] T aa4bb234488c470a9ef680243cd09522 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.040099 13275 raft_consensus.cc:385] T aa4bb234488c470a9ef680243cd09522 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:35.040385 13275 raft_consensus.cc:740] T aa4bb234488c470a9ef680243cd09522 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3517f3498e5e4e9cb5cbfa239e4999e3, State: Initialized, Role: FOLLOWER
I20260504 13:57:35.041112 13267 tablet_bootstrap.cc:492] T aa4bb234488c470a9ef680243cd09522 P 7c95ca5995cc4cbba624fd3ea6367902: Bootstrap starting.
I20260504 13:57:35.042441 13275 consensus_queue.cc:260] T aa4bb234488c470a9ef680243cd09522 P 3517f3498e5e4e9cb5cbfa239e4999e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.047935 13272 ts_tablet_manager.cc:1434] T 8896d8df10394e30ac15f3b04625b13b P 910fbf5d51f843609f5caf25e52beae6: Time spent starting tablet: real 0.039s user 0.011s sys 0.005s
I20260504 13:57:35.048995 13267 tablet_bootstrap.cc:654] T aa4bb234488c470a9ef680243cd09522 P 7c95ca5995cc4cbba624fd3ea6367902: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:35.055475 13275 ts_tablet_manager.cc:1434] T aa4bb234488c470a9ef680243cd09522 P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20260504 13:57:35.057173 12864 heartbeater.cc:499] Master 127.8.238.253:35717 was elected leader, sending a full tablet report...
I20260504 13:57:35.066064 13275 tablet_bootstrap.cc:492] T 8896d8df10394e30ac15f3b04625b13b P 3517f3498e5e4e9cb5cbfa239e4999e3: Bootstrap starting.
I20260504 13:57:35.071826 13272 tablet_bootstrap.cc:492] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6: Bootstrap starting.
I20260504 13:57:35.077994 13272 tablet_bootstrap.cc:654] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:35.082309 13275 tablet_bootstrap.cc:654] T 8896d8df10394e30ac15f3b04625b13b P 3517f3498e5e4e9cb5cbfa239e4999e3: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:35.090037 13275 tablet_bootstrap.cc:492] T 8896d8df10394e30ac15f3b04625b13b P 3517f3498e5e4e9cb5cbfa239e4999e3: No bootstrap required, opened a new log
I20260504 13:57:35.090462 13275 ts_tablet_manager.cc:1403] T 8896d8df10394e30ac15f3b04625b13b P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.004s
I20260504 13:57:35.093386 13267 tablet_bootstrap.cc:492] T aa4bb234488c470a9ef680243cd09522 P 7c95ca5995cc4cbba624fd3ea6367902: No bootstrap required, opened a new log
I20260504 13:57:35.093140 13275 raft_consensus.cc:359] T 8896d8df10394e30ac15f3b04625b13b P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } }
I20260504 13:57:35.093792 13267 ts_tablet_manager.cc:1403] T aa4bb234488c470a9ef680243cd09522 P 7c95ca5995cc4cbba624fd3ea6367902: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.016s
I20260504 13:57:35.093793 13275 raft_consensus.cc:385] T 8896d8df10394e30ac15f3b04625b13b P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:35.094062 13275 raft_consensus.cc:740] T 8896d8df10394e30ac15f3b04625b13b P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3517f3498e5e4e9cb5cbfa239e4999e3, State: Initialized, Role: FOLLOWER
I20260504 13:57:35.095474 13275 consensus_queue.cc:260] T 8896d8df10394e30ac15f3b04625b13b P 3517f3498e5e4e9cb5cbfa239e4999e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } }
I20260504 13:57:35.096460 13267 raft_consensus.cc:359] T aa4bb234488c470a9ef680243cd09522 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.097213 13267 raft_consensus.cc:385] T aa4bb234488c470a9ef680243cd09522 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:35.097512 13267 raft_consensus.cc:740] T aa4bb234488c470a9ef680243cd09522 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c95ca5995cc4cbba624fd3ea6367902, State: Initialized, Role: FOLLOWER
I20260504 13:57:35.103741 13267 consensus_queue.cc:260] T aa4bb234488c470a9ef680243cd09522 P 7c95ca5995cc4cbba624fd3ea6367902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.106060 13272 tablet_bootstrap.cc:492] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6: No bootstrap required, opened a new log
I20260504 13:57:35.106503 13272 ts_tablet_manager.cc:1403] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.002s
I20260504 13:57:35.108125 13275 ts_tablet_manager.cc:1434] T 8896d8df10394e30ac15f3b04625b13b P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260504 13:57:35.109230 13275 tablet_bootstrap.cc:492] T 2de1fe3c24ba40088ccb83968db0cf76 P 3517f3498e5e4e9cb5cbfa239e4999e3: Bootstrap starting.
I20260504 13:57:35.109834 13267 ts_tablet_manager.cc:1434] T aa4bb234488c470a9ef680243cd09522 P 7c95ca5995cc4cbba624fd3ea6367902: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20260504 13:57:35.112736 13267 tablet_bootstrap.cc:492] T e436ef80b9d845c0a313fd7e95229bd1 P 7c95ca5995cc4cbba624fd3ea6367902: Bootstrap starting.
I20260504 13:57:35.117650 13275 tablet_bootstrap.cc:654] T 2de1fe3c24ba40088ccb83968db0cf76 P 3517f3498e5e4e9cb5cbfa239e4999e3: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:35.119648 13272 raft_consensus.cc:359] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.120338 13272 raft_consensus.cc:385] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:35.120675 13272 raft_consensus.cc:740] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910fbf5d51f843609f5caf25e52beae6, State: Initialized, Role: FOLLOWER
I20260504 13:57:35.121465 13272 consensus_queue.cc:260] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.124439 13272 ts_tablet_manager.cc:1434] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6: Time spent starting tablet: real 0.018s user 0.005s sys 0.002s
I20260504 13:57:35.125298 13272 tablet_bootstrap.cc:492] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6: Bootstrap starting.
I20260504 13:57:35.139652 13272 tablet_bootstrap.cc:654] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:35.141197 13267 tablet_bootstrap.cc:654] T e436ef80b9d845c0a313fd7e95229bd1 P 7c95ca5995cc4cbba624fd3ea6367902: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:35.153362 13275 tablet_bootstrap.cc:492] T 2de1fe3c24ba40088ccb83968db0cf76 P 3517f3498e5e4e9cb5cbfa239e4999e3: No bootstrap required, opened a new log
I20260504 13:57:35.153913 13275 ts_tablet_manager.cc:1403] T 2de1fe3c24ba40088ccb83968db0cf76 P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent bootstrapping tablet: real 0.045s user 0.018s sys 0.000s
I20260504 13:57:35.155400 13272 tablet_bootstrap.cc:492] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6: No bootstrap required, opened a new log
I20260504 13:57:35.155872 13272 ts_tablet_manager.cc:1403] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.007s
I20260504 13:57:35.156668 13275 raft_consensus.cc:359] T 2de1fe3c24ba40088ccb83968db0cf76 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.157331 13275 raft_consensus.cc:385] T 2de1fe3c24ba40088ccb83968db0cf76 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:35.157625 13275 raft_consensus.cc:740] T 2de1fe3c24ba40088ccb83968db0cf76 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3517f3498e5e4e9cb5cbfa239e4999e3, State: Initialized, Role: FOLLOWER
I20260504 13:57:35.158283 13275 consensus_queue.cc:260] T 2de1fe3c24ba40088ccb83968db0cf76 P 3517f3498e5e4e9cb5cbfa239e4999e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.162283 13275 ts_tablet_manager.cc:1434] T 2de1fe3c24ba40088ccb83968db0cf76 P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260504 13:57:35.163151 13275 tablet_bootstrap.cc:492] T e436ef80b9d845c0a313fd7e95229bd1 P 3517f3498e5e4e9cb5cbfa239e4999e3: Bootstrap starting.
I20260504 13:57:35.168781 13267 tablet_bootstrap.cc:492] T e436ef80b9d845c0a313fd7e95229bd1 P 7c95ca5995cc4cbba624fd3ea6367902: No bootstrap required, opened a new log
I20260504 13:57:35.169188 13267 ts_tablet_manager.cc:1403] T e436ef80b9d845c0a313fd7e95229bd1 P 7c95ca5995cc4cbba624fd3ea6367902: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.004s
I20260504 13:57:35.174132 13272 raft_consensus.cc:359] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.174932 13272 raft_consensus.cc:385] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:35.175096 13275 tablet_bootstrap.cc:654] T e436ef80b9d845c0a313fd7e95229bd1 P 3517f3498e5e4e9cb5cbfa239e4999e3: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:35.175365 13272 raft_consensus.cc:740] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910fbf5d51f843609f5caf25e52beae6, State: Initialized, Role: FOLLOWER
I20260504 13:57:35.176223 13272 consensus_queue.cc:260] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.180015 13267 raft_consensus.cc:359] T e436ef80b9d845c0a313fd7e95229bd1 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.180663 13267 raft_consensus.cc:385] T e436ef80b9d845c0a313fd7e95229bd1 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:35.180918 13267 raft_consensus.cc:740] T e436ef80b9d845c0a313fd7e95229bd1 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c95ca5995cc4cbba624fd3ea6367902, State: Initialized, Role: FOLLOWER
I20260504 13:57:35.181746 13267 consensus_queue.cc:260] T e436ef80b9d845c0a313fd7e95229bd1 P 7c95ca5995cc4cbba624fd3ea6367902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.184533 13272 ts_tablet_manager.cc:1434] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260504 13:57:35.185161 13267 ts_tablet_manager.cc:1434] T e436ef80b9d845c0a313fd7e95229bd1 P 7c95ca5995cc4cbba624fd3ea6367902: Time spent starting tablet: real 0.016s user 0.006s sys 0.002s
I20260504 13:57:35.188413 13272 tablet_bootstrap.cc:492] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6: Bootstrap starting.
I20260504 13:57:35.194789 13275 tablet_bootstrap.cc:492] T e436ef80b9d845c0a313fd7e95229bd1 P 3517f3498e5e4e9cb5cbfa239e4999e3: No bootstrap required, opened a new log
I20260504 13:57:35.195225 13275 ts_tablet_manager.cc:1403] T e436ef80b9d845c0a313fd7e95229bd1 P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.007s
I20260504 13:57:35.197939 13275 raft_consensus.cc:359] T e436ef80b9d845c0a313fd7e95229bd1 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.198488 13275 raft_consensus.cc:385] T e436ef80b9d845c0a313fd7e95229bd1 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:35.198730 13275 raft_consensus.cc:740] T e436ef80b9d845c0a313fd7e95229bd1 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3517f3498e5e4e9cb5cbfa239e4999e3, State: Initialized, Role: FOLLOWER
I20260504 13:57:35.199318 13275 consensus_queue.cc:260] T e436ef80b9d845c0a313fd7e95229bd1 P 3517f3498e5e4e9cb5cbfa239e4999e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.201162 13275 ts_tablet_manager.cc:1434] T e436ef80b9d845c0a313fd7e95229bd1 P 3517f3498e5e4e9cb5cbfa239e4999e3: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260504 13:57:35.204584 13272 tablet_bootstrap.cc:654] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6: Neither blocks nor log segments found. Creating new log.
I20260504 13:57:35.214231 13272 tablet_bootstrap.cc:492] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6: No bootstrap required, opened a new log
I20260504 13:57:35.214700 13272 ts_tablet_manager.cc:1403] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.004s
I20260504 13:57:35.217548 13272 raft_consensus.cc:359] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.218292 13272 raft_consensus.cc:385] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260504 13:57:35.218638 13272 raft_consensus.cc:740] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910fbf5d51f843609f5caf25e52beae6, State: Initialized, Role: FOLLOWER
I20260504 13:57:35.219391 13272 consensus_queue.cc:260] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.229110 13272 ts_tablet_manager.cc:1434] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6: Time spent starting tablet: real 0.014s user 0.000s sys 0.005s
I20260504 13:57:35.248390 13237 raft_consensus.cc:493] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:35.248878 13237 raft_consensus.cc:515] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } }
I20260504 13:57:35.250765 13237 leader_election.cc:290] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229), 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331)
I20260504 13:57:35.252326 12675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8896d8df10394e30ac15f3b04625b13b" candidate_uuid: "7c95ca5995cc4cbba624fd3ea6367902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910fbf5d51f843609f5caf25e52beae6" is_pre_election: true
I20260504 13:57:35.252537 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8896d8df10394e30ac15f3b04625b13b" candidate_uuid: "7c95ca5995cc4cbba624fd3ea6367902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" is_pre_election: true
I20260504 13:57:35.252950 12675 raft_consensus.cc:2468] T 8896d8df10394e30ac15f3b04625b13b P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c95ca5995cc4cbba624fd3ea6367902 in term 0.
I20260504 13:57:35.253084 12818 raft_consensus.cc:2468] T 8896d8df10394e30ac15f3b04625b13b P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c95ca5995cc4cbba624fd3ea6367902 in term 0.
I20260504 13:57:35.254184 12901 leader_election.cc:304] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3517f3498e5e4e9cb5cbfa239e4999e3, 7c95ca5995cc4cbba624fd3ea6367902; no voters:
I20260504 13:57:35.255401 13237 raft_consensus.cc:2804] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:57:35.255733 13237 raft_consensus.cc:493] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:57:35.256027 13237 raft_consensus.cc:3060] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:57:35.262300 13237 raft_consensus.cc:515] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } }
I20260504 13:57:35.264878 12675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8896d8df10394e30ac15f3b04625b13b" candidate_uuid: "7c95ca5995cc4cbba624fd3ea6367902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "910fbf5d51f843609f5caf25e52beae6"
I20260504 13:57:35.265427 12675 raft_consensus.cc:3060] T 8896d8df10394e30ac15f3b04625b13b P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:57:35.273343 12675 raft_consensus.cc:2468] T 8896d8df10394e30ac15f3b04625b13b P 910fbf5d51f843609f5caf25e52beae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c95ca5995cc4cbba624fd3ea6367902 in term 1.
I20260504 13:57:35.274381 12900 leader_election.cc:304] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c95ca5995cc4cbba624fd3ea6367902, 910fbf5d51f843609f5caf25e52beae6; no voters:
I20260504 13:57:35.275928 13294 raft_consensus.cc:2804] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:57:35.277240 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8896d8df10394e30ac15f3b04625b13b" candidate_uuid: "7c95ca5995cc4cbba624fd3ea6367902" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3"
I20260504 13:57:35.277773 12818 raft_consensus.cc:3060] T 8896d8df10394e30ac15f3b04625b13b P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:57:35.282214 13294 raft_consensus.cc:697] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [term 1 LEADER]: Becoming Leader. State: Replica: 7c95ca5995cc4cbba624fd3ea6367902, State: Running, Role: LEADER
I20260504 13:57:35.282979 13294 consensus_queue.cc:237] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } }
I20260504 13:57:35.284173 12818 raft_consensus.cc:2468] T 8896d8df10394e30ac15f3b04625b13b P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c95ca5995cc4cbba624fd3ea6367902 in term 1.
I20260504 13:57:35.285569 13237 leader_election.cc:290] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [CANDIDATE]: Term 1 election: Requested vote from peers 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229), 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331)
W20260504 13:57:35.374560 12752 connection.cc:570] server connection from 127.0.0.1:39496 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:35.374702 12899 connection.cc:570] server connection from 127.0.0.1:41360 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:35.375324 12753 connection.cc:570] client connection to 127.8.238.253:35717 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:35.375312 12901 connection.cc:570] client connection to 127.8.238.253:35717 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:35.375481 13023 connection.cc:570] client connection to 127.8.238.253:35717 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:35.377020 13023 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.8.238.253:35717)
W20260504 13:57:35.378007 13023 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.8.238.253:35717)
W20260504 13:57:35.378921 13023 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.238.253:35717)
W20260504 13:57:35.379734 12610 connection.cc:570] server connection from 127.0.0.1:58062 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:35.380290 12610 connection.cc:570] client connection to 127.8.238.253:35717 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260504 13:57:35.384711 13089 negotiation.cc:336] Failed RPC negotiation. Trace:
0504 13:57:35.376997 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.8.238.253:35717 (local address 127.8.238.194:44143)
0504 13:57:35.381771 (+ 4774us) negotiation.cc:107] Waiting for socket to connect
0504 13:57:35.381818 (+ 47us) client_negotiation.cc:175] Beginning negotiation
0504 13:57:35.382058 (+ 240us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0504 13:57:35.383980 (+ 1922us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":4111}
W20260504 13:57:35.386795 13159 negotiation.cc:336] Failed RPC negotiation. Trace:
0504 13:57:35.380433 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.8.238.253:35717 (local address 127.8.238.195:36949)
0504 13:57:35.382048 (+ 1615us) negotiation.cc:107] Waiting for socket to connect
0504 13:57:35.382090 (+ 42us) client_negotiation.cc:175] Beginning negotiation
0504 13:57:35.382315 (+ 225us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0504 13:57:35.385920 (+ 3605us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":933}
W20260504 13:57:35.389760 13023 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.8.238.253:35717)
I20260504 13:57:35.431001 13282 raft_consensus.cc:493] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:35.431538 13282 raft_consensus.cc:515] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.435648 13282 leader_election.cc:290] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331), 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223)
I20260504 13:57:35.451579 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa4bb234488c470a9ef680243cd09522" candidate_uuid: "910fbf5d51f843609f5caf25e52beae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" is_pre_election: true
I20260504 13:57:35.452207 12818 raft_consensus.cc:2468] T aa4bb234488c470a9ef680243cd09522 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910fbf5d51f843609f5caf25e52beae6 in term 0.
I20260504 13:57:35.453917 13303 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6dc832d2f7f748dbb8e3ba2f73ec98a4)
I20260504 13:57:35.454413 12610 leader_election.cc:304] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3517f3498e5e4e9cb5cbfa239e4999e3, 910fbf5d51f843609f5caf25e52beae6; no voters:
I20260504 13:57:35.454622 13303 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:35.455385 13282 raft_consensus.cc:2804] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:57:35.455703 13282 raft_consensus.cc:493] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:57:35.455996 13282 raft_consensus.cc:3060] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:57:35.461445 13303 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:57:35.464741 13162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
I20260504 13:57:35.466452 13282 raft_consensus.cc:515] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.468560 12966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa4bb234488c470a9ef680243cd09522" candidate_uuid: "910fbf5d51f843609f5caf25e52beae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c95ca5995cc4cbba624fd3ea6367902" is_pre_election: true
I20260504 13:57:35.469146 12966 raft_consensus.cc:2468] T aa4bb234488c470a9ef680243cd09522 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910fbf5d51f843609f5caf25e52beae6 in term 0.
I20260504 13:57:35.469432 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa4bb234488c470a9ef680243cd09522" candidate_uuid: "910fbf5d51f843609f5caf25e52beae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3"
I20260504 13:57:35.469946 12818 raft_consensus.cc:3060] T aa4bb234488c470a9ef680243cd09522 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:57:35.471536 13282 leader_election.cc:290] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [CANDIDATE]: Term 1 election: Requested vote from peers 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331), 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223)
I20260504 13:57:35.473779 12966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa4bb234488c470a9ef680243cd09522" candidate_uuid: "910fbf5d51f843609f5caf25e52beae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c95ca5995cc4cbba624fd3ea6367902"
I20260504 13:57:35.474287 12966 raft_consensus.cc:3060] T aa4bb234488c470a9ef680243cd09522 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Advancing to term 1
W20260504 13:57:35.475699 13162 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:57:35.476298 13162 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:57:35.476493 12818 raft_consensus.cc:2468] T aa4bb234488c470a9ef680243cd09522 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910fbf5d51f843609f5caf25e52beae6 in term 1.
I20260504 13:57:35.476670 13162 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:35.477463 13303 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:57:35.481631 13282 raft_consensus.cc:493] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:35.481669 12610 leader_election.cc:304] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3517f3498e5e4e9cb5cbfa239e4999e3, 910fbf5d51f843609f5caf25e52beae6; no voters:
I20260504 13:57:35.482175 13282 raft_consensus.cc:515] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.482605 12966 raft_consensus.cc:2468] T aa4bb234488c470a9ef680243cd09522 P 7c95ca5995cc4cbba624fd3ea6367902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910fbf5d51f843609f5caf25e52beae6 in term 1.
I20260504 13:57:35.484638 13282 leader_election.cc:290] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331), 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223)
I20260504 13:57:35.484516 13307 raft_consensus.cc:2804] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:57:35.485513 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2de1fe3c24ba40088ccb83968db0cf76" candidate_uuid: "910fbf5d51f843609f5caf25e52beae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" is_pre_election: true
I20260504 13:57:35.485615 13307 raft_consensus.cc:697] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [term 1 LEADER]: Becoming Leader. State: Replica: 910fbf5d51f843609f5caf25e52beae6, State: Running, Role: LEADER
I20260504 13:57:35.485678 12966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2de1fe3c24ba40088ccb83968db0cf76" candidate_uuid: "910fbf5d51f843609f5caf25e52beae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c95ca5995cc4cbba624fd3ea6367902" is_pre_election: true
I20260504 13:57:35.486083 12818 raft_consensus.cc:2468] T 2de1fe3c24ba40088ccb83968db0cf76 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910fbf5d51f843609f5caf25e52beae6 in term 0.
I20260504 13:57:35.486238 12966 raft_consensus.cc:2468] T 2de1fe3c24ba40088ccb83968db0cf76 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910fbf5d51f843609f5caf25e52beae6 in term 0.
I20260504 13:57:35.486512 13307 consensus_queue.cc:237] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.487320 12610 leader_election.cc:304] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3517f3498e5e4e9cb5cbfa239e4999e3, 910fbf5d51f843609f5caf25e52beae6; no voters:
I20260504 13:57:35.489269 13282 raft_consensus.cc:2804] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:57:35.489637 13282 raft_consensus.cc:493] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:57:35.490049 13282 raft_consensus.cc:3060] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:57:35.499855 13282 raft_consensus.cc:515] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.504272 13282 leader_election.cc:290] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [CANDIDATE]: Term 1 election: Requested vote from peers 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331), 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223)
I20260504 13:57:35.505616 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2de1fe3c24ba40088ccb83968db0cf76" candidate_uuid: "910fbf5d51f843609f5caf25e52beae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3"
I20260504 13:57:35.505657 12966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2de1fe3c24ba40088ccb83968db0cf76" candidate_uuid: "910fbf5d51f843609f5caf25e52beae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c95ca5995cc4cbba624fd3ea6367902"
I20260504 13:57:35.506157 12818 raft_consensus.cc:3060] T 2de1fe3c24ba40088ccb83968db0cf76 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:57:35.506201 12966 raft_consensus.cc:3060] T 2de1fe3c24ba40088ccb83968db0cf76 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:57:35.512349 12966 raft_consensus.cc:2468] T 2de1fe3c24ba40088ccb83968db0cf76 P 7c95ca5995cc4cbba624fd3ea6367902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910fbf5d51f843609f5caf25e52beae6 in term 1.
I20260504 13:57:35.513434 12610 leader_election.cc:304] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c95ca5995cc4cbba624fd3ea6367902, 910fbf5d51f843609f5caf25e52beae6; no voters:
I20260504 13:57:35.515074 12818 raft_consensus.cc:2468] T 2de1fe3c24ba40088ccb83968db0cf76 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910fbf5d51f843609f5caf25e52beae6 in term 1.
I20260504 13:57:35.515372 13282 raft_consensus.cc:2804] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:57:35.515800 13282 raft_consensus.cc:697] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [term 1 LEADER]: Becoming Leader. State: Replica: 910fbf5d51f843609f5caf25e52beae6, State: Running, Role: LEADER
I20260504 13:57:35.516527 13282 consensus_queue.cc:237] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.864883 13282 raft_consensus.cc:493] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:35.865391 13282 raft_consensus.cc:515] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.867268 13282 leader_election.cc:290] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331), 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223)
I20260504 13:57:35.868049 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e436ef80b9d845c0a313fd7e95229bd1" candidate_uuid: "910fbf5d51f843609f5caf25e52beae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" is_pre_election: true
I20260504 13:57:35.868373 12966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e436ef80b9d845c0a313fd7e95229bd1" candidate_uuid: "910fbf5d51f843609f5caf25e52beae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c95ca5995cc4cbba624fd3ea6367902" is_pre_election: true
I20260504 13:57:35.868572 12818 raft_consensus.cc:2468] T e436ef80b9d845c0a313fd7e95229bd1 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910fbf5d51f843609f5caf25e52beae6 in term 0.
I20260504 13:57:35.868968 12966 raft_consensus.cc:2468] T e436ef80b9d845c0a313fd7e95229bd1 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 910fbf5d51f843609f5caf25e52beae6 in term 0.
I20260504 13:57:35.869426 12610 leader_election.cc:304] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3517f3498e5e4e9cb5cbfa239e4999e3, 910fbf5d51f843609f5caf25e52beae6; no voters:
I20260504 13:57:35.870151 13282 raft_consensus.cc:2804] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260504 13:57:35.870591 13282 raft_consensus.cc:493] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260504 13:57:35.870919 13282 raft_consensus.cc:3060] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:57:35.877226 13282 raft_consensus.cc:515] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:35.879024 13282 leader_election.cc:290] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [CANDIDATE]: Term 1 election: Requested vote from peers 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331), 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223)
I20260504 13:57:35.879766 12818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e436ef80b9d845c0a313fd7e95229bd1" candidate_uuid: "910fbf5d51f843609f5caf25e52beae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3"
I20260504 13:57:35.880218 12818 raft_consensus.cc:3060] T e436ef80b9d845c0a313fd7e95229bd1 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:57:35.880108 12966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e436ef80b9d845c0a313fd7e95229bd1" candidate_uuid: "910fbf5d51f843609f5caf25e52beae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c95ca5995cc4cbba624fd3ea6367902"
I20260504 13:57:35.880640 12966 raft_consensus.cc:3060] T e436ef80b9d845c0a313fd7e95229bd1 P 7c95ca5995cc4cbba624fd3ea6367902 [term 0 FOLLOWER]: Advancing to term 1
I20260504 13:57:35.886469 12818 raft_consensus.cc:2468] T e436ef80b9d845c0a313fd7e95229bd1 P 3517f3498e5e4e9cb5cbfa239e4999e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910fbf5d51f843609f5caf25e52beae6 in term 1.
I20260504 13:57:35.887558 12966 raft_consensus.cc:2468] T e436ef80b9d845c0a313fd7e95229bd1 P 7c95ca5995cc4cbba624fd3ea6367902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 910fbf5d51f843609f5caf25e52beae6 in term 1.
I20260504 13:57:35.887616 12610 leader_election.cc:304] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3517f3498e5e4e9cb5cbfa239e4999e3, 910fbf5d51f843609f5caf25e52beae6; no voters:
I20260504 13:57:35.888406 13282 raft_consensus.cc:2804] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260504 13:57:35.889638 13282 raft_consensus.cc:697] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [term 1 LEADER]: Becoming Leader. State: Replica: 910fbf5d51f843609f5caf25e52beae6, State: Running, Role: LEADER
I20260504 13:57:35.892089 13282 consensus_queue.cc:237] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 } } peers { permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 } } peers { permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 } }
I20260504 13:57:36.093734 13312 consensus_queue.cc:1048] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" member_type: VOTER last_known_addr { host: "127.8.238.193" port: 35229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:57:36.160974 13314 consensus_queue.cc:1048] T 8896d8df10394e30ac15f3b04625b13b P 7c95ca5995cc4cbba624fd3ea6367902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:57:36.214695 13320 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6dc832d2f7f748dbb8e3ba2f73ec98a4)
I20260504 13:57:36.215195 13320 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:36.218118 13320 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:57:36.226007 13162 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:57:36.226621 13162 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:57:36.227308 13162 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:36.229077 13320 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:57:36.305615 13282 consensus_queue.cc:1048] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:57:36.333182 13307 consensus_queue.cc:1048] T aa4bb234488c470a9ef680243cd09522 P 910fbf5d51f843609f5caf25e52beae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:57:36.389639 9147 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu with pid 13080
W20260504 13:57:36.423780 13024 connection.cc:570] client connection to 127.8.238.252:35825 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260504 13:57:36.480430 13307 consensus_queue.cc:1048] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:57:36.497094 13282 consensus_queue.cc:1048] T 2de1fe3c24ba40088ccb83968db0cf76 P 910fbf5d51f843609f5caf25e52beae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:57:36.838670 13282 consensus_queue.cc:1048] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" member_type: VOTER last_known_addr { host: "127.8.238.194" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:57:36.852010 13282 consensus_queue.cc:1048] T e436ef80b9d845c0a313fd7e95229bd1 P 910fbf5d51f843609f5caf25e52beae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" member_type: VOTER last_known_addr { host: "127.8.238.195" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260504 13:57:37.165630 9147 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
/tmp/dist-test-taskcMKItV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.252:35825
--webserver_interface=127.8.238.252
--webserver_port=35067
--builtin_ntp_servers=127.8.238.212:41493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--hive_metastore_uris=thrift://127.0.0.1:39899
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260504 13:57:37.508074 13339 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260504 13:57:37.508672 13339 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260504 13:57:37.509156 13339 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260504 13:57:37.544224 13339 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260504 13:57:37.544452 13339 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260504 13:57:37.544755 13339 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260504 13:57:37.545007 13339 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260504 13:57:37.545257 13339 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260504 13:57:37.545502 13339 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260504 13:57:37.585844 13339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.238.212:41493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:39899
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.238.252:35825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.238.252
--webserver_port=35067
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 04 May 2026 13:43:23 UTC on e7f111948823
build id 11743
TSAN enabled
I20260504 13:57:37.587327 13339 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260504 13:57:37.588819 13339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260504 13:57:37.599351 13350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:57:37.600102 13351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260504 13:57:39.003955 13349 debug-util.cc:398] Leaking SignalData structure 0x7b080002ecc0 after lost signal to thread 13339
W20260504 13:57:39.320168 13349 kernel_stack_watchdog.cc:198] Thread 13339 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260504 13:57:39.325052 13339 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.725s user 0.620s sys 1.039s
W20260504 13:57:39.325527 13339 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.726s user 0.620s sys 1.039s
W20260504 13:57:39.326519 13352 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1725 milliseconds
W20260504 13:57:39.327490 13353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260504 13:57:39.327533 13339 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260504 13:57:39.328689 13339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260504 13:57:39.331209 13339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260504 13:57:39.332582 13339 hybrid_clock.cc:648] HybridClock initialized: now 1777903059332509 us; error 74 us; skew 500 ppm
I20260504 13:57:39.333360 13339 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260504 13:57:39.339546 13339 webserver.cc:492] Webserver started at http://127.8.238.252:35067/ using document root <none> and password file <none>
I20260504 13:57:39.340442 13339 fs_manager.cc:362] Metadata directory not provided
I20260504 13:57:39.340687 13339 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260504 13:57:39.348593 13339 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260504 13:57:39.353454 13370 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260504 13:57:39.354522 13339 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260504 13:57:39.354852 13339 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data,/tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal
uuid: "26e2451840464331a15d5291c3f09034"
format_stamp: "Formatted at 2026-05-04 13:57:21 on dist-test-slave-f9wt"
I20260504 13:57:39.356805 13339 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260504 13:57:39.409288 13339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260504 13:57:39.410719 13339 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260504 13:57:39.411154 13339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260504 13:57:39.482102 13339 rpc_server.cc:307] RPC server started. Bound to: 127.8.238.252:35825
I20260504 13:57:39.482180 13422 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.238.252:35825 every 8 connection(s)
I20260504 13:57:39.485001 13339 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskcMKItV/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1777902952773179-9147-0/minicluster-data/master-2/data/info.pb
I20260504 13:57:39.494561 9147 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskcMKItV/build/tsan/bin/kudu as pid 13339
I20260504 13:57:39.495628 13423 sys_catalog.cc:263] Verifying existing consensus state
I20260504 13:57:39.502162 13423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: Bootstrap starting.
I20260504 13:57:39.523706 13423 log.cc:826] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: Log is configured to *not* fsync() on all Append() calls
I20260504 13:57:39.580295 13423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: Bootstrap replayed 1/1 log segments. Stats: ops{read=22 overwritten=0 applied=22 ignored=0} inserts{seen=14 ignored=0} mutations{seen=16 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260504 13:57:39.581121 13423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: Bootstrap complete.
I20260504 13:57:39.598368 13423 raft_consensus.cc:359] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:39.600440 13423 raft_consensus.cc:740] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e2451840464331a15d5291c3f09034, State: Initialized, Role: FOLLOWER
I20260504 13:57:39.601226 13423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 22, Last appended: 3.22, Last appended by leader: 22, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:39.603880 13427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } } }
I20260504 13:57:39.604548 13427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: This master's current role is: FOLLOWER
I20260504 13:57:39.604638 13423 sys_catalog.cc:565] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [sys.catalog]: configured and running, proceeding with master startup.
I20260504 13:57:39.637247 13441 catalog_manager.cc:1269] Loaded cluster ID: 8a9361cd032446ef9af554d6cfd5f3e8
I20260504 13:57:39.637603 13441 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: loading cluster ID for follower catalog manager: success
I20260504 13:57:39.646790 13441 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: acquiring CA information for follower catalog manager: success
I20260504 13:57:39.653038 13441 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260504 13:57:39.657279 13011 heartbeater.cc:344] Connected to a master server at 127.8.238.252:35825
I20260504 13:57:39.636850 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:57:39.680951 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:57:39.683645 13387 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7c95ca5995cc4cbba624fd3ea6367902" instance_seqno: 1777903050251388) as {username='slave'} at 127.8.238.195:33205; Asking this server to re-register.
I20260504 13:57:39.686291 13011 heartbeater.cc:461] Registering TS with master...
I20260504 13:57:39.692586 13387 ts_manager.cc:194] Registered new tserver with Master: 7c95ca5995cc4cbba624fd3ea6367902 (127.8.238.195:36223)
I20260504 13:57:39.773595 12721 heartbeater.cc:344] Connected to a master server at 127.8.238.252:35825
I20260504 13:57:39.777338 13387 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "910fbf5d51f843609f5caf25e52beae6" instance_seqno: 1777903046020339) as {username='slave'} at 127.8.238.193:56271; Asking this server to re-register.
I20260504 13:57:39.778529 12721 heartbeater.cc:461] Registering TS with master...
I20260504 13:57:39.781399 13387 ts_manager.cc:194] Registered new tserver with Master: 910fbf5d51f843609f5caf25e52beae6 (127.8.238.193:35229)
I20260504 13:57:39.893693 12863 heartbeater.cc:344] Connected to a master server at 127.8.238.252:35825
I20260504 13:57:39.896937 13387 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "3517f3498e5e4e9cb5cbfa239e4999e3" instance_seqno: 1777903048076284) as {username='slave'} at 127.8.238.194:35745; Asking this server to re-register.
I20260504 13:57:39.898089 12863 heartbeater.cc:461] Registering TS with master...
I20260504 13:57:39.900815 13387 ts_manager.cc:194] Registered new tserver with Master: 3517f3498e5e4e9cb5cbfa239e4999e3 (127.8.238.194:40331)
I20260504 13:57:40.202348 13448 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:40.202921 13448 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:40.205991 13448 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:57:40.209988 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:57:40.214095 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:57:40.214736 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:57:40.215044 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:40.215683 13448 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:57:40.924549 13454 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:40.924969 13454 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:40.926772 13454 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:57:40.933984 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:57:40.934566 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:57:40.934844 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:40.935418 13454 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:57:41.896098 13466 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:41.896481 13466 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:41.897946 13466 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:57:41.902400 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:57:41.902992 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:57:41.903337 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:41.903878 13466 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:57:41.995518 9147 master-stress-test.cc:577] Killing test thread 1/10
I20260504 13:57:43.141800 13484 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:43.142118 13484 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:43.143438 13484 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:57:43.148927 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:57:43.149518 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:57:43.149804 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:43.150352 13484 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:57:44.683794 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:57:44.684162 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:57:44.886394 13515 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:44.886834 13515 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:44.888381 13515 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:57:44.892773 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:57:44.893290 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:57:44.893580 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:44.894112 13515 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:57:46.349728 13539 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:46.350112 13539 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:46.351663 13539 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:57:46.353854 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260504 13:57:46.356827 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:57:46.357601 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:57:46.357924 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:46.358479 13539 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:57:49.687642 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:57:49.688068 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:57:49.905447 13592 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:49.905972 13592 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:49.908063 13592 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:57:49.913440 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:57:49.913968 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:57:49.914290 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:49.914886 13592 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:57:51.148964 13610 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:57:51.149369 13610 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:57:51.150780 13610 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:57:51.155644 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:57:51.156270 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:57:51.156550 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:57:51.157017 13610 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:57:54.690578 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:57:54.690958 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 13:57:58.191972 13007 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5fa0 after lost signal to thread 12875
W20260504 13:57:58.192871 13007 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4920 after lost signal to thread 13010
I20260504 13:57:59.694885 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:57:59.695433 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:58:03.425141 13782 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:58:03.425693 13782 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:58:03.427114 13782 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:58:03.429865 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260504 13:58:03.433068 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:58:03.433712 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:58:03.434026 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:58:03.434528 13782 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:58:04.697897 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:58:04.698273 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:58:04.866940 13808 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:58:04.867625 13808 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:58:04.869819 13808 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:58:04.876394 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:58:04.876971 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:58:04.877321 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:58:04.877954 13808 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:58:09.701526 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:58:09.701956 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:58:14.704562 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:58:14.704947 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:58:19.708433 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:58:19.708871 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:58:22.223450 14053 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:58:22.224004 14053 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:58:22.225842 14053 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:58:22.228296 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260504 13:58:22.231392 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:58:22.232035 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:58:22.232328 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:58:22.232857 14053 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:58:24.711627 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:58:24.711983 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:58:28.233687 14145 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:58:28.234192 14145 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:58:28.235951 14145 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:58:28.238260 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260504 13:58:28.241317 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:58:28.241959 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:58:28.242275 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:58:28.242899 14145 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:58:29.714709 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:58:29.715097 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 13:58:31.066661 13012 heartbeater.cc:646] Failed to heartbeat to 127.8.238.253:35717 (118 consecutive failures): Network error: Failed to ping master at 127.8.238.253:35717: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
W20260504 13:58:31.138579 12723 heartbeater.cc:646] Failed to heartbeat to 127.8.238.254:38403 (133 consecutive failures): Network error: Failed to ping master at 127.8.238.254:38403: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:58:31.214808 12865 heartbeater.cc:646] Failed to heartbeat to 127.8.238.254:38403 (131 consecutive failures): Network error: Failed to ping master at 127.8.238.254:38403: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:58:31.304425 13419 debug-util.cc:398] Leaking SignalData structure 0x7b08000a82e0 after lost signal to thread 13343
W20260504 13:58:31.304989 13419 debug-util.cc:398] Leaking SignalData structure 0x7b08000a9f40 after lost signal to thread 13422
I20260504 13:58:34.761317 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:58:34.761809 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:58:39.765213 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:58:39.765647 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:58:43.308189 14352 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:58:43.308701 14352 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:58:43.310523 14352 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:58:43.312770 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260504 13:58:43.315999 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:58:43.316569 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:58:43.316864 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:58:43.317437 14352 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:58:44.768381 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:58:44.768770 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 13:58:49.053984 12859 debug-util.cc:398] Leaking SignalData structure 0x7b08000dba40 after lost signal to thread 12733
W20260504 13:58:49.054937 12859 debug-util.cc:398] Leaking SignalData structure 0x7b08000cf3a0 after lost signal to thread 12862
I20260504 13:58:49.773206 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:58:49.773707 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:58:50.656491 14451 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:58:50.656960 14451 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:58:50.658631 14451 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:58:50.661911 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260504 13:58:50.665477 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:58:50.666260 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:58:50.666671 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:58:50.667516 14451 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:58:51.372390 14468 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:58:51.372758 14468 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:58:51.374163 14468 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:58:51.378602 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:58:51.379261 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:58:51.379551 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:58:51.380018 14468 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:58:54.776309 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:58:54.776672 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:58:57.240726 14543 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:58:57.241117 14543 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:58:57.242579 14543 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:58:57.245128 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260504 13:58:57.248370 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:58:57.248811 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:58:57.249080 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:58:57.249570 14543 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260504 13:58:57.350685 12717 debug-util.cc:398] Leaking SignalData structure 0x7b08000d9bc0 after lost signal to thread 12582
W20260504 13:58:57.351522 12717 debug-util.cc:398] Leaking SignalData structure 0x7b08000d25a0 after lost signal to thread 12720
I20260504 13:58:59.779677 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:58:59.780083 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:59:04.782544 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:59:04.782881 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:59:09.786386 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:59:09.786774 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:59:11.935519 14737 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:59:11.936012 14737 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:59:11.937635 14737 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:59:11.939833 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260504 13:59:11.942054 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:59:11.942782 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:59:11.943037 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:59:11.943567 14737 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260504 13:59:12.037755 13419 debug-util.cc:398] Leaking SignalData structure 0x7b08000500e0 after lost signal to thread 13343
W20260504 13:59:12.038283 13419 debug-util.cc:398] Leaking SignalData structure 0x7b08000ab0e0 after lost signal to thread 13422
I20260504 13:59:12.618175 14751 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:59:12.618571 14751 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:59:12.620040 14751 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:59:12.624704 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:59:12.625231 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:59:12.625500 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:59:12.625967 14751 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:59:14.815017 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:59:14.815411 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:59:19.818698 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:59:19.819085 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:59:20.794339 14873 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:59:20.794787 14873 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:59:20.796265 14873 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:59:20.798377 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260504 13:59:20.800802 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:59:20.801493 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:59:20.801734 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:59:20.802210 14873 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:59:24.821610 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:59:24.821960 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 13:59:29.492691 13007 debug-util.cc:398] Leaking SignalData structure 0x7b08000cc3c0 after lost signal to thread 12875
W20260504 13:59:29.494076 13007 debug-util.cc:398] Leaking SignalData structure 0x7b08000b56c0 after lost signal to thread 13010
I20260504 13:59:29.826004 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:59:29.826573 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 13:59:31.539115 13013 heartbeater.cc:646] Failed to heartbeat to 127.8.238.254:38403 (251 consecutive failures): Network error: Failed to ping master at 127.8.238.254:38403: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:59:31.639870 12723 heartbeater.cc:646] Failed to heartbeat to 127.8.238.254:38403 (253 consecutive failures): Network error: Failed to ping master at 127.8.238.254:38403: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:59:31.646639 12864 heartbeater.cc:646] Failed to heartbeat to 127.8.238.253:35717 (238 consecutive failures): Network error: Failed to ping master at 127.8.238.253:35717: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:59:32.680692 15036 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:59:32.681213 15036 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:59:32.682912 15036 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:59:32.685568 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260504 13:59:32.688342 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:59:32.689080 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:59:32.689339 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:59:32.689893 15036 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260504 13:59:33.925714 13007 debug-util.cc:398] Leaking SignalData structure 0x7b08000c5440 after lost signal to thread 12875
W20260504 13:59:33.926548 13007 debug-util.cc:398] Leaking SignalData structure 0x7b0800041760 after lost signal to thread 13010
I20260504 13:59:34.829195 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:59:34.829618 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:59:39.722064 13377 maintenance_manager.cc:419] P 26e2451840464331a15d5291c3f09034: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033366
I20260504 13:59:39.802152 13376 maintenance_manager.cc:643] P 26e2451840464331a15d5291c3f09034: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.079s user 0.038s sys 0.009s Metrics: {"bytes_written":9481,"cfile_init":1,"drs_written":1,"lbm_read_time_us":231,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2379,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":14,"thread_start_us":476,"threads_started":1,"wal-append.queue_time_us":800}
I20260504 13:59:39.804171 13377 maintenance_manager.cc:419] P 26e2451840464331a15d5291c3f09034: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 847 bytes on disk
I20260504 13:59:39.805636 13376 maintenance_manager.cc:643] P 26e2451840464331a15d5291c3f09034: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":234,"lbm_reads_lt_1ms":4}
I20260504 13:59:39.832792 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:59:39.833189 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 13:59:44.789191 13419 debug-util.cc:398] Leaking SignalData structure 0x7b0800076240 after lost signal to thread 13343
W20260504 13:59:44.789858 13419 debug-util.cc:398] Leaking SignalData structure 0x7b0800070f00 after lost signal to thread 13422
I20260504 13:59:44.856922 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:59:44.857268 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 13:59:45.799722 12859 debug-util.cc:398] Leaking SignalData structure 0x7b08000bee60 after lost signal to thread 12733
W20260504 13:59:45.800603 12859 debug-util.cc:398] Leaking SignalData structure 0x7b08000c52c0 after lost signal to thread 12862
I20260504 13:59:49.143072 15282 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:59:49.143654 15282 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:59:49.146055 15282 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:59:49.148564 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260504 13:59:49.151440 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:59:49.152184 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:59:49.152537 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:59:49.153117 15282 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:59:49.860076 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:59:49.860497 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 13:59:51.525308 15322 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:59:51.525731 15322 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:59:51.527673 15322 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:59:51.534292 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:59:51.534884 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:59:51.535147 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:59:51.535679 15322 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:59:54.892671 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:59:54.893006 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 13:59:55.027249 13419 debug-util.cc:398] Leaking SignalData structure 0x7b0800070b20 after lost signal to thread 13343
W20260504 13:59:55.027817 13419 debug-util.cc:398] Leaking SignalData structure 0x7b08000ab2e0 after lost signal to thread 13422
I20260504 13:59:56.143527 15405 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 13:59:56.143879 15405 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 13:59:56.145147 15405 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 13:59:56.147460 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260504 13:59:56.150061 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 13:59:56.150497 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 13:59:56.150735 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 13:59:56.151247 15405 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 13:59:59.896638 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 13:59:59.897053 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:00:04.899783 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:00:04.900138 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:00:05.932135 15565 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 14:00:05.932683 15565 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 14:00:05.934685 15565 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 14:00:05.937201 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260504 14:00:05.940836 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:00:05.942443 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 14:00:05.942898 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 14:00:05.943645 15565 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 14:00:09.903298 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:00:09.903733 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:00:14.906219 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:00:14.906599 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:00:19.909786 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:00:19.910192 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:00:21.120561 15816 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 14:00:21.121068 15816 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 14:00:21.123036 15816 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 14:00:21.125257 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260504 14:00:21.128350 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:00:21.128932 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 14:00:21.129213 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 14:00:21.129789 15816 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 14:00:22.824086 15847 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 14:00:22.824545 15847 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 14:00:22.826449 15847 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 14:00:22.831202 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:00:22.831830 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 14:00:22.832122 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 14:00:22.832636 15847 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 14:00:24.912410 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:00:24.912781 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 14:00:29.145730 13007 debug-util.cc:398] Leaking SignalData structure 0x7b08000d8220 after lost signal to thread 12875
W20260504 14:00:29.147149 13007 debug-util.cc:398] Leaking SignalData structure 0x7b080003f940 after lost signal to thread 13010
I20260504 14:00:29.916368 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:00:29.916807 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 14:00:31.955019 13013 heartbeater.cc:646] Failed to heartbeat to 127.8.238.254:38403 (371 consecutive failures): Network error: Failed to ping master at 127.8.238.254:38403: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:00:32.088699 12864 heartbeater.cc:646] Failed to heartbeat to 127.8.238.253:35717 (358 consecutive failures): Network error: Failed to ping master at 127.8.238.253:35717: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
W20260504 14:00:32.101228 12723 heartbeater.cc:646] Failed to heartbeat to 127.8.238.254:38403 (373 consecutive failures): Network error: Failed to ping master at 127.8.238.254:38403: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:00:34.218761 12717 debug-util.cc:398] Leaking SignalData structure 0x7b08000b3880 after lost signal to thread 12582
W20260504 14:00:34.220414 12717 debug-util.cc:398] Leaking SignalData structure 0x7b08000dc6e0 after lost signal to thread 12720
I20260504 14:00:34.919305 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:00:34.919677 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:00:39.922855 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:00:39.923326 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:00:41.251993 16174 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 14:00:41.252578 16174 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 14:00:41.255023 16174 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 14:00:41.258046 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260504 14:00:41.261194 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:00:41.261771 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 14:00:41.262038 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 14:00:41.262569 16174 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 14:00:44.942281 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:00:44.942649 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 14:00:45.494392 13419 debug-util.cc:398] Leaking SignalData structure 0x7b0800077ce0 after lost signal to thread 13343
W20260504 14:00:45.494917 13419 debug-util.cc:398] Leaking SignalData structure 0x7b0800077840 after lost signal to thread 13422
I20260504 14:00:49.945968 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:00:49.946485 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:00:54.949059 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:00:54.949457 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:00:56.174420 16426 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 14:00:56.174882 16426 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 14:00:56.176565 16426 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 14:00:56.178838 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260504 14:00:56.182550 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:00:56.182971 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 14:00:56.183254 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 14:00:56.183784 16426 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 14:00:59.952893 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:00:59.953336 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:01:04.956665 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:01:04.957212 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:01:08.609306 16637 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 14:01:08.610061 16637 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 14:01:08.611663 16637 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 14:01:08.614614 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260504 14:01:08.617620 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:01:08.618080 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 14:01:08.618337 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 14:01:08.618845 16637 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 14:01:09.960170 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:01:09.960639 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:01:14.963299 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:01:14.963670 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:01:19.966928 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:01:19.967442 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:01:22.252561 16865 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 14:01:22.253124 16865 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 14:01:22.255025 16865 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 14:01:22.257448 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260504 14:01:22.260720 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:01:22.261276 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 14:01:22.261531 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 14:01:22.262077 16865 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 14:01:24.970084 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:01:24.970460 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:01:29.973691 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:01:29.974128 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 14:01:32.347855 13013 heartbeater.cc:646] Failed to heartbeat to 127.8.238.254:38403 (491 consecutive failures): Network error: Failed to ping master at 127.8.238.254:38403: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:01:32.470377 12865 heartbeater.cc:646] Failed to heartbeat to 127.8.238.254:38403 (491 consecutive failures): Network error: Failed to ping master at 127.8.238.254:38403: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:01:32.486244 12723 heartbeater.cc:646] Failed to heartbeat to 127.8.238.254:38403 (493 consecutive failures): Network error: Failed to ping master at 127.8.238.254:38403: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
I20260504 14:01:34.986308 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:01:34.986779 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 14:01:36.006203 12859 debug-util.cc:398] Leaking SignalData structure 0x7b08000da040 after lost signal to thread 12733
W20260504 14:01:36.007033 12859 debug-util.cc:398] Leaking SignalData structure 0x7b08000d1ee0 after lost signal to thread 12862
I20260504 14:01:37.208487 17114 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 14:01:37.208974 17114 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 14:01:37.210510 17114 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 14:01:37.212703 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260504 14:01:37.216506 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:01:37.217240 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 14:01:37.217487 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 14:01:37.217954 17114 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 14:01:39.989977 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:01:39.990425 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:01:44.992964 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:01:44.993333 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:01:49.996917 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:01:49.997372 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:01:53.195485 17380 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 14:01:53.195919 17380 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 14:01:53.198024 17380 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 14:01:53.200554 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260504 14:01:53.203234 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:01:53.203799 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 14:01:53.204025 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 14:01:53.204569 17380 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 14:01:54.999756 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:01:55.000084 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:01:56.448487 17435 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 14:01:56.448832 17435 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 14:01:56.450366 17435 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 14:01:56.455648 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:01:56.456110 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 14:01:56.456377 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 14:01:56.456897 17435 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 14:02:00.003580 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:02:00.004034 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 14:02:02.720455 13007 debug-util.cc:398] Leaking SignalData structure 0x7b08000c51c0 after lost signal to thread 12875
W20260504 14:02:02.721424 13007 debug-util.cc:398] Leaking SignalData structure 0x7b08000d8680 after lost signal to thread 13010
I20260504 14:02:05.007043 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:02:05.007439 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:02:10.010661 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:02:10.011061 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:02:15.013732 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:02:15.014128 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:02:16.046020 17762 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 14:02:16.046547 17762 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 14:02:16.048400 17762 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 14:02:16.050846 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260504 14:02:16.053819 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:02:16.054366 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 14:02:16.054636 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 14:02:16.055151 17762 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260504 14:02:20.017266 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:02:20.017716 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:02:23.012365 17880 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 14:02:23.012892 17880 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 14:02:23.014430 17880 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 14:02:23.017128 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260504 14:02:23.021299 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:02:23.021786 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 14:02:23.022063 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 14:02:23.022574 17880 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260504 14:02:24.857717 13419 debug-util.cc:398] Leaking SignalData structure 0x7b0800070180 after lost signal to thread 13343
W20260504 14:02:24.858266 13419 debug-util.cc:398] Leaking SignalData structure 0x7b08000777e0 after lost signal to thread 13422
I20260504 14:02:25.045914 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:02:25.046311 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:02:29.353652 17989 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260504 14:02:29.354179 17989 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc3d546912c9427aaca93bdafc33dd6d" member_type: VOTER last_known_addr { host: "127.8.238.254" port: 38403 } } peers { permanent_uuid: "6dc832d2f7f748dbb8e3ba2f73ec98a4" member_type: VOTER last_known_addr { host: "127.8.238.253" port: 35717 } } peers { permanent_uuid: "26e2451840464331a15d5291c3f09034" member_type: VOTER last_known_addr { host: "127.8.238.252" port: 35825 } }
I20260504 14:02:29.355681 17989 leader_election.cc:290] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403), 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717)
W20260504 14:02:29.358258 13374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260504 14:02:29.361495 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc3d546912c9427aaca93bdafc33dd6d (127.8.238.254:38403): Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:02:29.362078 13374 leader_election.cc:336] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6dc832d2f7f748dbb8e3ba2f73ec98a4 (127.8.238.253:35717): Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
I20260504 14:02:29.362304 13374 leader_election.cc:304] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e2451840464331a15d5291c3f09034; no voters: 6dc832d2f7f748dbb8e3ba2f73ec98a4, bc3d546912c9427aaca93bdafc33dd6d
I20260504 14:02:29.362768 17989 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 26e2451840464331a15d5291c3f09034 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260504 14:02:29.986913 12717 debug-util.cc:398] Leaking SignalData structure 0x7b08000ceb60 after lost signal to thread 12582
W20260504 14:02:29.987803 12717 debug-util.cc:398] Leaking SignalData structure 0x7b08000cda20 after lost signal to thread 12720
I20260504 14:02:30.048942 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:02:30.049340 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260504 14:02:32.324456 12717 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2d80 after lost signal to thread 12582
W20260504 14:02:32.325249 12717 debug-util.cc:398] Leaking SignalData structure 0x7b0800041b40 after lost signal to thread 12720
W20260504 14:02:32.763994 13013 heartbeater.cc:646] Failed to heartbeat to 127.8.238.254:38403 (611 consecutive failures): Network error: Failed to ping master at 127.8.238.254:38403: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
W20260504 14:02:32.868085 12722 heartbeater.cc:646] Failed to heartbeat to 127.8.238.253:35717 (600 consecutive failures): Network error: Failed to ping master at 127.8.238.253:35717: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
W20260504 14:02:32.947633 12865 heartbeater.cc:646] Failed to heartbeat to 127.8.238.254:38403 (611 consecutive failures): Network error: Failed to ping master at 127.8.238.254:38403: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111)
I20260504 14:02:35.052105 13339 master_runner.cc:186] Error getting master registration for 127.8.238.254:38403: Network error: Client connection negotiation failed: client connection to 127.8.238.254:38403: connect: Connection refused (error 111),
I20260504 14:02:35.052469 13339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260504 14:02:35.183627 13377 maintenance_manager.cc:419] P 26e2451840464331a15d5291c3f09034: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 847 bytes on disk
I20260504 14:02:35.194419 13376 maintenance_manager.cc:643] P 26e2451840464331a15d5291c3f09034: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.010s user 0.009s sys 0.000s Metrics: {"dirs.queue_time_us":639,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":211,"thread_start_us":774,"threads_started":2,"wal-append.queue_time_us":670}
W20260504 14:02:35.641602 13023 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
W20260504 14:02:35.642135 13023 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
W20260504 14:02:35.642478 13023 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
W20260504 14:02:35.642762 13023 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
F20260504 14:02:35.657280 13029 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_398fe552fc9a43ac8875babd6d5001fc on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20260504 14:02:35.659513 13030 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2dc3318442a1464eb3c4db4f57a86c83 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
*** Aborted at 1777903355 (unix time) try "date -d @1777903355" if you are using GNU date ***
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000023bb) received by PID 9147 (TID 0x7f7d116fe700) from PID 9147; stack trace: ***
@ 0x2de460 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2e0943 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f7d2300b980 (unknown) at ??:0
@ 0x7f7d1e1cce87 gsignal at ??:0
@ 0x7f7d1e1ce7f1 abort at ??:0
@ 0x2e4456 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f7d21264da4 google::LogMessage::SendToLog() at ??:0
@ 0x7f7d21265950 google::LogMessage::Flush() at ??:0
@ 0x7f7d2126aa8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x360594 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x38e45e _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3846
@ 0x38e311 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x38da5b std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2de320 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
F20260504 14:02:35.659513 13030 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_2dc3318442a1464eb3c4db4f57a86c83 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)F000F20260504 14:02:35.669019 13031 master-stress-test.cc:297] 000F20260504 14:02:35.671950 13032 master-stress-test.cc:297] 000F20260504 14:02:35.713757 13033 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.8.238.254:38403,127.8.238.253:35717,127.8.238.252:35825, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.8.238.253:35717: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f7d230006db start_thread at ??:0
@ 0x7f7d1e2af61f clone at ??:0